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EXECUTIVE  SUMMARY 


RT-19,  Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering,  is  a  research  study 
whose  goal  is  to  understand  the  impact  on  student  learning  of  and  career  interest  in  Systems 
Engineering  (SE)  through  a  set  of  diverse  capstone  courses  that  expose  students  to  authentic 
Department  of  Defense  (DoD)  problems  and  engage  them  in  learning  and  practice  of  systems 
engineering.  SE  Capstone  courses  were  developed  and  piloted  during  the  2010-11  academic  year  (and 
beyond)  in  eight  civilian  universities  and  six  military  institutions  affiliated  with  the  Systems  Engineering 
Research  Center  (SERC).  The  strategic  goal  addressed  by  this  research  is  to  better  understand  how 
differing  course  designs,  structures,  materials,  instructional  practices,  and  other  inputs,  such  as  the 
involvement  of  DoD  and  industry  mentors,  impact  student  learning  and  career  interest  in  SE.  This 
research  explored  methods  and  approaches  to  augment  the  SE  workforce  for  future  DoD  and  related 
industry  workforce  needs  in  order  to  inform  future  investments  for  the  purpose  of  institutionalizing  and 
scaling  up  effective  methods.  This  research  encompassed  a  20-month,  three-phase  effort  from  March  1, 
2010  to  October  31,  2011,  including  planning,  course  implementation,  and  analysis.  Institutions  were 
selected  for  participation  through  a  competitive  application  process  based  on  a  set  of  criteria  developed 
in  consultation  with  the  sponsor,  and  partners  were  awarded  a  subcontract  of  approximately  $200,000 
for  development,  implementation,  analysis,  and  reporting  on  their  SE  Capstone  project. 

According  to  final  reports  submitted  by  principal  investigators,  330  and  257  students  participated  in  RT- 
19-sponsored  SE  Capstone  courses  in  the  fall  2010  and  spring  2011  semesters,  respectively.  Many 
institutions  enrolled  the  same  students  for  both  semesters,  but  a  few,  such  as  the  University  of 
Maryland,  enrolled  a  new  cohort  of  students  in  the  spring,  bringing  the  total  number  of  students 
impacted  to  more  than  360.  Approximately  half  were  undergraduates,  of  whom  the  majority  were 
fourth  year  seniors.  Of  the  graduate  students,  most  were  first  year  students,  with  small  percentage 
post-graduates  participating  in  roles  such  as  project  manager. 

Four  topic  areas  illustrating  authentic  DoD  problems  were  presented  for  student  teams'  projects. 
Problem  area  #1,  low-cost,  low-power  computer  solutions  (see  Table  2  for  more  complete  description), 
was  the  most  heavily  subscribed  topic,  with  more  than  half  the  projects  addressing  this  problem  area. 
Problem  areas  were  selected,  in  part,  based  on  expertise  of  participating  faculty  and  institutional 
resources,  and  on  availability  of  DoD  and  local  experts.  Institutions  organized  their  teams  in  different 
ways:  the  most  common  structure  included  several  teams  working  on  several  different  design  problems. 

A  majority  of  the  universities  relied  on  the  expertise  of  systems  engineering  faculty  to  lead  or  contribute 
to  the  conceptualization,  development,  and  implementation  of  the  course,  but  many  other  faculty  were 
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involved  as  well,  particularly  from  mechanical  engineering  and  computer  science  departments.  At  11 
institutions,  faculty  came  from  at  least  three  separate  engineering  disciplines,  literally  embodying  the 
multi-disciplinarity  of  an  SE  team.  Nearly  two-thirds  of  the  14  projects  were  planned  and  implemented 
by  teams  of  two  or  three  faculty  members,  but  four  projects  included  four  or  more  faculty.  Only  one 
institution  (US  Naval  Academy)  developed  a  Capstone  course  that  was  planned  and  taught  by  a  single 
faculty  member. 

The  research  team  gathered  the  following  data  in  order  to  analyze  the  impact  of  the  SE  Capstone  project 
on  student  learning  of  SE,  student  interest  in  SE  careers,  and  student  awareness/interest  in  authentic 
DoD  problems:  pre/post  student  surveys;  pre/post  case  study  analysis  by  students;  and  student  blog 
posts.  In  addition,  this  report  also  contains  input  gathered  from  the  July  SE  Capstone  conference,  review 
and  analysis  of  final  reports  submitted  by  principal  investigators,  as  well  as  papers,  publications,  and 
posters  developed  by  faculty,  researchers,  and  students. 

Many  faculty  used  customized  assessments  and  other  means  (e.g.,  student  participation  in 
competitions)  to  assess  student  outcomes  (See  Appendix  B  for  description  of  course  materials/student 
deliverables  and  internal  assessments).  Through  semantic  analysis  of  students'  constructed  responses 
on  definitions  of  systems  engineering  (administered  in  pre-  and  post-course  surveys)  as  compared  to 
two  expert  definitions,  larger  gains  were  observed  for  undergraduates  and  students  with  no  prior  SE 
experience  than  for  students  who  self-identified  with  prior  SE  knowledge.  Students  with  no  prior  SE 
experience  not  only  showed  larger  gains,  but  also  they  ended  with  a  slightly  higher  percentage  than  the 
group  as  a  whole. 

The  research  team  used  an  analytic  rubric  to  measure  changes  in  the  level  of  complexity  of  student 
thinking  using  systems  engineering  knowledge  from  pre-  to  post-course  on  the  case  study  analysis  of  the 
Bradley  Fighting  Vehicle.  A  life  cycle  model  was  used  to  map  units  of  competency  from  the  SPRDE- 
SE/PSE  Competency  Model  into  lower  (definitional)  or  higher  (development,  deployment)  categories  of 
analysis  (Sage,  2000,  p.  166).  For  the  entire  set  of  matched  responses,  statistically  significant  increases 
were  recorded  for  all  categories  of  competencies  combined,  and  for  categories  B  and  C  (denoting  more 
sophisticated  reasoning).  Students  with  prior  SE  experience  had  higher  initial  and  final  scores,  but  the 
difference  between  the  two  groups'  (students  with  and  without  prior  SE  experience)  was  smaller  by  the 
post-test,  suggesting  that  the  SE  Capstone  courses  positively  impacted  student  learning  of  SE, 
particularly  for  those  students  without  SE  experience.  The  increases  were  statistically  significant  for  both 
groups  of  students. 

Finally,  weekly  posts  to  weblogs  ("blogs")  and  a  final  post  were  required  of  all  teams  in  order  to  provide 
qualitative  data  and  insights  into  the  changes  in  the  level  of  complexity  of  students'  thinking  about  SE  as 
applied  to  their  Capstone  project.  Blogs  were  used  in  a  variety  of  ways  by  partner  institutions; 
therefore,  generalizations  about  the  RT-19  student  population  cannot  be  made  from  this  data  source. 
Student  blog  posts  described  phases  of  the  SE  design  process;  included  project  artifacts  and  media  files; 
and  described  challenges  teams  encountered  during  the  project.  These  included  such  issues  as  making 
design  tradeoffs;  providing  adequate  security  for  their  (wireless)  products;  relying  too  much  on 
knowledge  or  technical  skills  of  one  team  member  with  a  specific  area  of  expertise;  setting  reasonable 
and  achievable  goals  for  product  design  within  a  school  year;  communication  challenges  in  an 
interdisciplinary  team  with  different  perspectives  and  varying  levels  of  expertise;  and  managing  time 
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and  design  constraints.  The  most  common  student  responses  about  the  most  challenging  aspects  of  the 
project  included  managing  the  dynamics  of  a  multi-disciplinary  group  and  communication  problems. 

Overall,  82%  of  responding  students  felt  their  group  produced  a  successful  product.  Of  those  who  did 
not  feel  their  projects  were  successful,  lack  of  resources  and  time  were  the  most  frequently  cited 
reasons. 

A  goal  of  the  SE  Capstone  courses  implemented  in  RT-19  was  to  increase  student  awareness  of  the 
diversity  of  problems  addressed  by  the  DoD.  From  pre-  to  post-survey,  changes  from  very  general  to 
more  specific  types  of  problems  identified  by  students,  including  greater  use  of  SE  terminology,  were 
observed.  The  problem  area  that  increased  the  most  in  students'  awareness  was  energy-related, 
particularly  energy  efficiency  and  green  energy,  while  the  area  that  decreased  the  most  was  weapons 
and  weapon  systems. 

Another  goal  of  SE  Capstone  courses  was  to  increase  student  interest  in:  SE  careers  generally;  SE 
careers  in  government;  and  SE  careers  in  industry.  Post-survey  means  for  the  entire  population  of 
matched  pre-/post-survey  responses  increased  in  all  three  categories,  although  these  increases  were 
not  statistically  significant.  For  those  students  with  prior  SE  experience,  Q1  (general)  and  Q3  (industry) 
increased,  while  the  mean  for  Q2  (government)  decreased.  For  the  matched  group  without  prior  SE 
experience,  the  means  for  all  three  questions  increased,  with  the  mean  for  Q3  (industry)  increasing  the 
most.  Further  analysis  of  students'  Likert  Scale  responses  show  more  subtle  differences  in  the  level  of 
interest  (from  low  to  high)  among  the  various  subgroups  analyzed.  Eighty  percent  of  students  who 
responded  to  an  open-ended  question  asking  whether  they  would  pursue  a  career  in  systems 
engineering  career  and,  if  so,  why,  stated  that  they  would,  and  many  indicated  this  would  be  some  time 
in  the  future  after  gaining  experience  in  their  chosen  engineering  discipline.  The  remaining  20  percent 
who  responded  they  would  not  consider  a  career  in  SE  listed  a  preference  for  pursuing  their  chosen 
engineering  career  as  the  main  reason.  Overall,  a  large  majority  of  students  who  answered  an  open- 
ended  question  about  the  applicability  of  systems  engineering  to  future  engineering  studies  and  plans 
(64  of  67)  agreed  that  SE  provided  a  useful  framework  and  broad  perspective  needed  to  manage 
complex  engineering  challenges. 

The  recruitment,  involvement,  and  impact  of  DoD  and  industry  mentors  is  an  aspect  of  the  SE  Capstone 
project  which  deserves  special  emphasis  in  the  analysis  of  the  overall  project  in  light  of  the  intensive 
efforts  made  to  connect  mentors  with  student  teams,  the  voluntary  nature  of  mentors'  roles,  and 
implications  for  sustainability  and  scaling  up.  All  Capstone  partner  institutions  had  a  DoD  mentor,  and 
about  half  had  additional  mentors.  Mentors  played  the  role  of  clients  as  well  as  technical  experts, 
guiding  students  toward  solutions.  Lack  of  role  definition  of  mentors  was  cited  as  problematic  by  Pis; 
preference  was  expressed  for  DoD  mentors  to  serve  as  clients.  Lack  of  and  late  start  of  mentor 
involvement  with  student  teams,  as  well  as  varying  levels  and  frequency  of  communication  between 
mentors  and  students  were  cited  as  challenges.  Beneficial  impacts  of  mentor  involvement  were 
reported  by  Pis  when  communication  was  frequent  and  specific,  particularly  in  the  case  of  design 
reviews.  Defense  prime  contractors  who  served  as  mentors  were  reported  to  provide  a  different 
perspective  than  DoD  mentors,  chiefly,  by  representing  "the  solution  viewpoint"  and  "saving  student 
teams  from  exploring  too  many  blind  alleys."  Student  rankings  of  DoD  and  industry  mentors'  usefulness 
in  learning  about  and  applying  SE  in  their  courses  were  in  the  mid-  to  low  range  for  all  but  one 
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institution,  as  compared  to  six  other  course  inputs  such  as  lectures  and  other  team  members.  In 
considering  sustaining  mentor  relationships  and  scaling  up,  it  will  be  important  to  examine  value  of 
mentors,  the  sustainable  features  of  mentor  relationships,  and  the  characteristics  of  particularly  strong 
mentor  relationships. 

Through  site  visits  to  SE  Capstone  universities  in  spring  2011,  a  team  of  sponsor  representatives 
identified  a  set  of  promising  practices— approaches  which  were  present  in  universities  where  students 
demonstrated  higher  levels  of  communication,  analysis,  and  awareness  of  the  SE  process  during  the  site 
visits.  Although  limitations  of  the  data  and  the  scope  of  RT-19  do  not  allow  for  analysis  of  correlations 
between  these  promising  practices  and  the  SE  Capstone  models  that  may  have  led  to  greater  student 
outcomes,  these  promising  practices  have  informed  the  research  being  undertaken  through  RT-19A,  the 
Pilot  for  Scaling  Up  and  Sustaining  Effective  SE  Capstone  Practices.  A  graphical  representation  of  the 
presence  (or  lack  thereof)  of  these  promising  practices  among  all  participating  RT-19  universities 
appears  as  Appendix  E. 

In  order  to  institutionalize  aspects  of  the  SE  Capstone  project  post-DoD  funding,  it  will  be  necessary  to 
address  the  critical  challenges  faced  by  faculty  who  are  responsible  for  implementing  these  projects. 
Some  challenges  identified  by  faculty  resulted  from  the  accelerated  startup  and  logistical  demands 
associated  with  a  new  course.  Here,  issues  such  as  recruitment,  material  and  assessment  development, 
coordination  of  schedules  among  students  in  different  engineering  disciplines  and  coordination  with 
external  mentors  were  commonly  cited  challenges.  Other  challenges  were  associated  with  establishing 
a  broad,  overarching  SE  framework  in  the  context  of  traditional  departmental  academic  structures. 
Course  expectations,  grading  policies,  and  formation  of  teams  representing  different  disciplines  were 
cited  as  issues,  as  were  negotiating  the  optimal  balance  between  SE  content  knowledge  and  discipline- 
specific  technical  expertise  among  students  and  faculty  and  identifying  manageable  project  scope  for 
the  given  instructional  period.  Lastly,  models  for  sustaining  and  institutionalizing  SE  Capstone  projects 
were  proposed,  including  fee-based  programs  in  which  students  work  on  a  problem  from  industry  or 
government  as  a  contractor. 

Findings  and  Recommendations: 

Limitations  in  the  data  and  the  many  approaches  and  variables  used  in  the  14  pilot  courses  prevent 
statistical  correlations  with  student  outcomes  and  "optimal"  course  designs.  However,  the  following 
summary  of  findings  are  grounded  in  data  collected  through  RT-19: 

Analysis  of  student  definitions  of  systems  engineering  showed  that  participating  students  were  able  to 
use  general  systems  engineering  terminology  almost  as  well  as  experts  but  that  they  still  had  some  way 
to  go  in  employing  more  technical  systems  engineering  language.  However,  those  with  the  most  to 
learn— undergraduates  and  those  with  no  prior  system  engineering  experience— improved  the  most, 
particularly  in  terms  of  technical  language. 

Analysis  of  the  Bradley  Fighting  Vehicle  case  study  showed  that  students  increased  in  their  ability  to 
identify  problems  that  mapped  to  specific  systems  engineering  competencies,  particularly  those  related 
to  the  technical  elements,  but  that  they  were  less  likely  to  mention  the  "soft"  competencies  like 
communication  and  leadership. 
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The  blogs,  where  used  well,  showed  students  working  through  the  phases  of  the  design  process  and 
struggling  with  various  technical  and  communication  issues  along  the  way. 

Students  enjoyed  the  real-world  nature  of  the  projects— both  in  terms  of  building  an  artifact  that  might 
be  used  and  in  terms  of  the  SE  project  context  (budget  constraints,  interdisciplinary  teams,  experts  as 
mentor)— and  that  they  appreciated  the  contribution  that  the  systems  engineering  perspective  brought 
to  their  work. 

SE  Capstone  courses  do  not  appear  to  have  had  a  major  impact  on  the  students'  immediate  career 
plans,  it  must  be  noted  that  many  had  their  immediate  post-college  plans  in  place  and  that  a  large 
majority  of  both  undergraduates  and  graduate  students  believed  that  they  might  choose  careers  in 
systems  engineering  sometime  in  the  future. 

Recommendations  for  future  implementations  and  future  research  include: 

1.  Develop  a  methodology  to  prioritize  and  rank  the  student  attributes  and  outcomes  most  likely  to 
meet  DoD  and  defense  industry  needs  in  the  near  term  (0-5  years)  and  longer  term. 

2.  Examine  the  presence,  depth,  and  characteristics  of  implementation  of  the  promising  practices 
through  case  study  analysis  (a  component  of  research  included  in  RT-19A);  correlate,  where 
possible,  to  the  highest  priority  student  attributes  described  in  (1),  above. 

3.  Distill  the  attributes  of  effective  DoD  and  industry  mentor  relationships  through  further  analysis  of 
"what  worked"  and  what  did  not.  Investigate  the  incentives  and  rewards  for  mentors  to  continue 
involvement  with  university  partners. 

4.  Make  very  explicit  the  goal  of  attracting  students  to  DoD  careers  in  systems  engineering  in 
coursework  and  other  communications;  provide  technical  assistance  and  other  materials  to  mentors 
and  faculty. 

5.  Leverage  the  experience  and  expertise  of  the  RT-19  and  RT-19A  to  build  and  expand  a  learning 
community  of  SE  Capstone  stakeholders  (engineering  institutions,  clients,  and  mentors). 

6.  Consider  piloting  new  approaches  to  sustain  the  SE  Capstone  project,  including  the  creation  of  an 
online  repository  of  potential  DoD  problem  areas  and  clients  along  with  a  "venture  fund"  that  would 
provide  small  grants  of  $5,000-$10,000  for  materials  and  access  to  DoD  problems  and  clients  for 
institutions  that  already  organize  Capstone  projects. 

7.  Publicize  in  relevant  professional  journals,  education  media,  and  the  general  media  the 
contributions  of  SE  Capstone  design  teams  to  the  development  of  solutions  critical  for  our  military 
and  our  nation's  security. 

8.  Conduct  a  longer-term  study  (1-5  years)  tracking  RT-19  participants  and  their  career  choices  and 
employment  trends. 
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1.0  INTRODUCTION 


1.1  PROJECT  OVERVIEW 

A  45%  growth  is  expected  in  SE  jobs  in  the  next  decade  and  there  have  been  numerous  studies  and 
workshops  that  have  highlighted  the  shortfalls  in  both  the  number  and  capability  of  the  SE  workforce 
(Rosato,  Braverman,  &  Jeffries,  2009).  The  July  2006  National  Defense  Industrial  Association  (NDIA)  Task 
Force  noted  among  the  top  five  SE  issues  the  lack  of  adequate,  qualified  SE  human  capital  resources 
within  government  and  industry  for  allocation  on  major  programs  (National  Defense  Industrial 
Association  SE  Division  Task  Group ,  2006).  In  the  July  2010  NDIA  white  paper  on  critical  SE  challenges, 
Issue  2  was  identified  as:  The  quantity  and  quality  ofSE  expertise  is  insufficient  to  meet  the  demands  of 
the  government  and  defense  industry,  and  further  outlined  certain  recommendations  to  build  SE 
expertise  and  capacity.  In  particular,  it  recommended  developing  SE  expertise  through  "role  definition, 
selection,  training,  career  incentives,  and  broadening  'systems  thinking'  into  other  disciplines,"  and 
made  a  number  of  specific  recommendations,  including  adding  an  introductory  course  in  SE  in  all 
undergraduate  engineering  and  technical  management  degree  programs;  and  working  with  major 
universities  to  recommend  SE  curricula  to  improve  consistency  across  programs  in  order  to  achieve 
standardization  of  skill  sets  for  graduates  (National  Defense  Industrial  Association  SE  Division,  2010). 
With  these  industry-wide  workforce  demands  challenging  the  systems  engineering  community,  RT-19 
was  conceptualized  and  designed  to  pilot  and  evaluate  approaches  to  ameliorating  these  shortages. 


1.2  RESEARCH  OBJECTIVES  AND  PROGRAM  GOALS 

Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering,  (referred  to  as  the  SE 
Capstone  Project),  aims  to  understand  the  methods  through  which  SE  learning  and  career  interest  may 
be  increased  among  undergraduate  and  graduate  engineering  students.  The  key  research  question  this 
program  was  designed  to  address  is: 

What  organization  of  course  work  (course  sequence,  course  materials,  faculty  characteristics,  student 
characteristics  and  other  inputs  and  activities)  leads  to  the  largest  student  gains  in  (1)  SE  learning;  (2) 
interest  in  SE  careers;  and  (3)  interest  in  DoD  problems  and  careers? 

This  research  was  conducted  in  the  context  of  14  "capstone"  courses,  in  most  cases  as  an  integrative 
culminating,  project-based  course  involving  teams  of  students  working  together  on  the  development  of 
a  product  or  prototype  that  addresses  a  real  DoD  need.  Implemented  as  pilot  courses  in  eight  civilian 
and  six  military  universities  affiliated  with  the  Systems  Engineering  Research  Center,  these  14 
institutions  piloted  methods,  materials,  and  approaches  to  create  new  courses  or  enhance  existing 
courses  to  embed,  infuse,  and  augment  SE  knowledge,  as  defined  by  the  Systems  Planning,  Research 
Development,  and  Engineering  (SPRDE)-SE  and  Program  Systems  Engineer  (PSE)  competency  model, 
known  as  the  SPRDE-SE/PSE  Competency  Model  (Table  1),  among  undergraduate  and  graduate 
students.  Participating  university  faculty  developed  new  course  materials  and  other  methods  and 
strategies  to  recruit  and  provide  substantive  SE  learning  experiences;  increase  exposure  to  authentic 
DoD  problems,  such  as  low-cost,  low-power  computing  devices,  expeditionary  assistance  kits, 
expeditionary  housing  systems,  and  immersive  training  technologies. 
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This  pilot  program  was  conducted  in  order  to  inform  the  development  of  a  national  scale-up  effort  that 
would  substantially  expand  the  number  and  capabilities  of  universities  that  could  produce  SE  graduates 
needed  for  the  DoD  and  related  defense  industry  workforce.  It  was  anticipated  a  portfolio  of  shareable 
course  materials,  assessment  instruments,  and  other  lessons  would  be  produced  in  order  to  accelerate 
the  adoption  of  effective  practices  and  materials  in  a  national  scale  up.  Analysis  of  student  data  from 
several  sources,  PI  reports,  input  from  sponsors'  site  visit  teams,  and  insights  gleaned  from  panels  and 
presentations  at  a  culminating  workshop  form  the  basis  for  the  content  of  this  final  report  and 
recommendations. 


Analytical 

(13) 

1 .  Technical  Basis  for  Cost 

2.  Modeling  and  Simulation 

3.  Safety  Assurance 

4.  Stakeholder  Requirements  Definition 
(Requirements  Development) 

5.  Requirements  Analysis  (Logical  Analysis) 

6.  Architectural  Design  (Design  Solution) 

7.  Implementation 

8.  Integration 

9.  Verification 

10.  Validation 

1 1 .  Transition 

12.  System  Assurance 

13.  Reliability,  Availability,  and  Maintainability 

Technical 

Management 

(12) 

14.  Decision  Analysis 

15.  Technical  Planning 

16.  Technical  Assessment 

17.  Configuration  Management 

18.  Requirements  Management 

19.  Risk  Management 

20.  Technical  Data  Management 

21.  Interface  Management 

22.  Software  Engineering 

23.  Acquisition 

24.  Systems  Engineering  Leadership 

25.  System  of  Systems 

Professional 

(4) 

26.  Communications 

27.  Problem  Solving 

28.  Strategic  Thinking 

29.  Professional  Ethics 

Table  1:  SPRDE-SE/PSE  Competency  Model 


The  14  pilot  universities  were  required  to  address  one  or  more  of  four  DoD  problem  areas  and  to 
produce  an  actual  product,  prototype,  or  other  artifact  to  demonstrate  their  learning. 
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DoD  Problem  Areas 

1 .  Low-cost,  low-power  computers  leveraging  open-source  technologies  and 
advanced  security  to  support  sustainable,  secure  collaboration; 

Portable,  renewable  power  generation,  storage,  and  distribution  to  support 
sustained  operations  in  austere  environments  and  reduce  dependency  on 
carbon-based  energy  sources;  Portable,  low-power  water  purification; 

2.  An  expeditionary  assistance  kit  around  low-cost,  efficient,  and  sustainable 
prototypes  such  as  solar  cookers,  small  and  transportable  shelters,  deployable 
information  and  communication  technologies,  water  purifiers,  and  renewable 
energies.  These  materials  would  be  packaged  in  mission-specific  HA/DR  kits  for 
partner  nation  use; 

3.  Develop  modular,  scalable,  expeditionary  housing  systems  that  possess  "green" 
electric  power  and  water  generation,  waste  and  wastewater  disposal,  hygiene, 
and  food  service  capabilities.  Systems  should  be  designed  to  blend  in  to 
natural/native  surroundings  and  with  minimal  footprint; 

4.  Continued  investigation  and  exploration  into  the  realm  of  the  possible  with  respect 
to  “Immersive”  training  technologies.  Objective  is  to  flood  the  training  audience 
environment  with  the  same  STIMULI  that  one  would  experience  during  actual 
mission  execution.  Where  possible  full  sensory  overload  is  desired  much  the 
same  as  experienced  in  combat.  Specific  S&T  areas  for  development 

Virtual  Human.  Successful  modeling  of  emotions,  speech  patterns,  cultural 
behaviors,  dialogue  and  gestures. 

Universal  Language  Model.  The  ability  for  trainees  to  seamlessly  converse 
with  the  Virtual  Human. 

Virtual  Character  Grab  Controls.  The  ability  for  exercise  controllers  to 
assume  control  of  virtual  characters. 

Automated  Programming.  Cognitive  learning  models  and  the  ability  for 
exercise  controllers  to  adjust  virtual/live  simulations. 

Low  Cost  wireless  personnel  sensors. 

Sensors  (i.e.,  lightweight  vests)  that  facilitate  physical  stimuli  (i.e.,  wounds, 
shots)  to  trainees. 


Table  2:  DoD  Problem  Areas 

1.3  RESEARCH  QUESTIONS  AND  METHODS 
Methodology:  Measurement  of  Student  Educational  Outcomes 

The  impact  of  the  variety  of  SE  Capstone  courses  on  the  student  outcomes  identified  above  was 
designed  to  be  measured  through  the  administration  of  three  "common"  assessments,  or  assessments 
required  of  all  SE  Capstone  student  participants.  These  were  designed  to  be  administered  at  the 
beginning  (pre-)  and  end  (post-)  of  their  Capstone  course  experience.  These  included: 
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1.  Pre/Post  Survey,  focused  on  student  knowledge  of  systems  engineering,  interest  in  systems 
engineering  careers,  and  awareness  of  DoD  problem  areas. 

2.  Pre/Post  Case  Study  Analysis  [Bradley  Fighting  Vehicle],  a  semantic  analysis  designed  to  capture 
growth  in  SE  approach/analysis 

3.  Student  Blogs  were  intended  to  provide  qualitative  evidence  of  the  progress  in  level  of 
sophistication  of  student  analysis. 

Also,  faculty  of  each  participating  institution  developed  customized  assessments  that  were  unique  to 
their  courses  using  diverse  instruments  such  as  those  listed  below: 

•  Comprehensive  Rubrics 

•  Student  Presentations/Briefings  for  design  reviews  and  Final  Project  Presentation 

•  Peer  Review 

•  Team  Reports 

A  chart  delineating  the  various  internal  assessments  used  by  each  partner  university  appears  as 
Appendix  B. 


1.4  PARTICIPATING  INSTITUTIONS  AND  SELECTION  CRITERIA 

A  request  for  proposals  was  issued  and  a  competitive  application  process  was  conducted  in  order  to 
select  SE  Capstone  partner  institutions.  An  independent  panel  of  SE  and  engineering  education  experts 
used  a  common  scoring  rubric  to  evaluate  11  proposals,  which  resulted  in  the  selection  of  eight  civilian 
institutions,  which  appear  in  Table  3.  A  separate  process  managed  by  the  Office  of  the  Assistant 
Secretary  of  Defense  for  Research  and  Engineering  (ASDR&E)  resulted  in  the  participation  of  four  service 
academies  working  under  the  direction  of  the  Naval  Postgraduate  School  and  Air  Force  Institute  of 
Technology,  bringing  the  total  number  to  14  partner  institutions. 


Partners 


Civilian  Universities 


1.  Auburn  University 

2.  Missouri  University  S  &.  T 


3.  Penn  State 

4.  Southern  Methodist  University 


5.  Stevens  Institute  of  Technology 

6.  University  of  Maryland 

7.  University  of  Virginia 


Service  Academies 


1.  Air  Force  Institute  of  Technology 


2.  Naval  Postgraduate  School 


3.  Air  Force  Academy 

4.  Military  Academy  —  West  Point 

5.  Coast  Guard  Academy 


6.  Naval  Academy 


afii 


i. 


SVtM  I \  >illl 


Table  3:  Partner  Institutions 
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1.5  TIMELINE 


The  following  is  a  brief  description  of  the  three  phases  of  the  RT19  project: 

Phase  1/Startup  (March  1 , 201 0-May  1 5,  201 0)  was  accelerated  in  order  to  provide  program 
requirements  and  executed  subcontracts  to  enable  partner  universities  to  develop  materials  and 
conduct  program  implementation  in  the  Fall  2010  academic  semester.  Two  universities  conducted  one- 
semester  Capstone  courses;  11  conducted  two-semester  courses;  and  one  (NPS)  organized  its  Capstone 
course  over  multiple  terms. 

Phase  2/Pilot  Implementation  (May  15,  2010-June  30,  2011)  Capstone  institutions  developed  course 
materials  and  assessment  instruments  (July-September  2010);  delivered  the  courses  (August  2010-May 
2011);  and  submitted  two  interim  reports  (July  2010  and  January  2011)  and  a  final  report  (June  2011). 
Some  variation  in  this  schedule  was  based  on  the  specific  calendar  for  classes  at  each  Capstone  Team 
member. 

Phase  3/Analysis,  Recommendations  &  Dissemination  (July  1 ,  201 1  -  October  31 , 201 1 )  This  phase 
coordinated  a  summative  workshop  for  all  RT-19  and  RT-19A  (the  follow-on  research)  constituents,  and 
conducted/compiled  the  analysis  of  results  of  student  assessments  and  other  data  and  artifacts  for 
submission  in  a  final  report. 

The  following  figure  shows  the  milestones  of  each  phase: 
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RT19  Project  Timeline 


Phase  I: 
Start  Up 

2.5  Months 


Phase  II:  Pilot  Implementation 


March  1  — 
May  15. 
2010 


13  5  Month: 


May  15.  2010- June  30. 2011 


Phase  III: 
Analysis 

4  Months 

July  1 -Oct  31, 
2011 


MAM 


MJ  JASONDJ  FMAMJ  JASO 


Pitot*  Submit 
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Figure  1:  RT19  Project  Timeline 


1.6  SUMMARY  OF  PARTICIPANTS  IMPACTED 


According  to  final  reports  submitted  by  principal  investigators,  330  and  257  students  participated  in  RT- 
19-sponsored  SE  Capstone  courses  in  the  fall  2010  and  spring  2011  semesters,  respectively.  Many 
institutions  enrolled  the  same  students  for  both  semesters,  but  a  few,  such  as  the  University  of 
Maryland,  enrolled  a  new  cohort  of  students  in  the  spring,  bringing  the  total  number  of  students 
impacted  to  more  than  360.  Approximately  half  were  undergraduates,  of  whom  the  majority  were 
fourth  year  seniors.  Of  the  graduate  students,  most  were  first  year  students,  with  small  percentage 
post-graduates  participating  in  roles  such  as  project  manager. 

While  the  total  number  of  undergraduates  and  graduate  students  was  nearly  equal  across  the  13 
institutions,  a  closer  look  at  differences  between  individual  institutions  shows  that  nearly  half  of 
the  13  institutions  (Penn  State,  UVA,  SMU,  CGA,  AFA,  and  West  Point]  were  comprised  entirely  of 
undergraduates.  Four  institutions  (Wayne  State,  AFIT,  NPS,  and  the  Naval  Academy]  enrolled 
graduate  students  (including  postgraduate  students]  and  the  remaining  three  (Auburn,  Missouri 
S&T,  and  Stevens]  enrolled  both  undergraduate  and  graduate  students.  However,  the  ratio  varied: 
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while  Auburn  had  mostly  graduate  and  postgraduate  students  (92  percent],  with  fairly  few 
undergraduates,  Stevens  had  a  2:3  ratio  of  undergraduates  to  graduate  students,  and  Missouri  S&T 
had  a  1:4  ratio. 


All  Institutions-  Student  Participants 


350 


Figure  2:  All  Institutions  -  Student  Participants 


1.7  FACULTY  INVOLVEMENT 

According  to  original  proposals  and  project  reports,  approximately  50  faculty  participated  in  the 
development,  delivery,  and/or  assessment  of  the  SE  Capstone  courses  across  the  14  participating 
institutions.  A  majority  of  the  universities  relied  on  the  expertise  of  systems  engineering  faculty  to  lead 
or  contribute  to  the  conceptualization,  development,  and  implementation  of  the  program,  but  many 
other  faculty  were  involved  as  well,  particularly  from  mechanical  engineering  and  computer  science.  At 
11  institutions,  faculty  came  from  at  least  three  separate  engineering  disciplines,  literally  embodying  the 
multi-disciplinary  character  of  a  systems  engineering  team.  In  the  following  graph,  percentages 
represent  the  percentage  of  the  14  pilot  universities  that  included  those  types  of  disciplinary  faculty  in 
the  RT-19  project. 
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Figure  3:  Faculty  Involvement 


Nearly  two-thirds  of  the  fourteen  projects  were  planned  and  implemented  by  teams  of  two  or  three 
faculty  members,  but  four  projects  included  four  or  more  faculty.  Only  one  institution  developed  a 
capstone  course  that  was  planned  and  taught  by  a  single  faculty  member 


Faculty  teams 

■  2  to  3  faculty  members 
per  Institution 

■  4  or  more 
□  1  faculty  member 


Figure  4:  Faculty  Teams 


1.8  DOD/INDUSTRY  MENTORS 

Approximately  30  DoD  and  industry  mentors  contributed  to  the  SE  Capstone  projects.  In  many  cases, 
mentors  were  recruited  from  the  project  kickoff  meeting  in  August  2010,  while  in  others,  institutions 
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tapped  their  own  networks  to  recruit  mentors.  In  some  institutions  mentors  played  multiple  roles, 
including  as  client  and  as  technical  advisor. 

PI  reports  included  the  following  mentor  contacts: 


University/Academy 

Mentors 

Auburn 

Advisory  board  (5  SE  professionals  from  govt,  and  industry) 

Industry  Mentor  (automotive  arena) 

PhD  TAs  (support  team) 

Missouri  S&T 

Boeing  company  engineers:  Dale  Waldo,  Louis  Pape,  Nancy 

Pendleton,  Robert  Simmons  and  Robert  Scheurer 

Office  of  Naval  Research:  Pete  Muller 

Penn  State 

DoD  Mentors:  Col.  Nancy  Grandy,  and  Mr.  Phil  Stockdale 

Southern  Methodist 

U.S.  Marine  Corps 

Office  of  Naval  Research:  Pete  Muller 

Stevens  Institute 

Naval  Surface  Warfare  Center:  Eric  Shields 

Red  Gate  Group,  Ltd:  Joseph  Barniak 

U  of  Maryland 

Lockheed  Martin:  Sandy  Friedenthal 

DoD  Mentors:  Dr.  David  Robie,  Kim  Watkins 

U  of  Virginia 

DoD  Mentor:  Bill  Campbell 

Northrop  Grumman  engineers 

Wayne  State 

Army  Shelter  Expert:  Claudia  Quigley 

ArmyTARDEC:  Dr.  PeteSchil 

Military  Academy 

SRI/Sarnoff:  Dr.  Rakesh  Kumar 

DoD  Mentors:  LTC  Joe  Nolan,  LTC  Chris  Vaughn  [Joint  Advanced 

Training  Technologies  Lab] 

Air  Force  Academy 

DoD  Mentors:  a  reserve  AF  Colonel,  a  retired  USMC  officer 

Table  4:  Mentors 

Eight  civilian  universities  and  two  service  academies  reported  working  with  mentors  as  shown  in  the 
table  above. 
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2.0  ANALYSIS  OF  SYSTEMS  ENGINEERING  STUDENT 

LEARNING  OUTCOMES 


RT-19-sponsored  SE  Capstone  courses  were  designed  to  increase  student: 

•  Understanding  of  the  discipline  of  systems  engineering 

•  Understanding  of  the  work  of  systems  engineers 

•  Facility  and  practice  of  SE  knowledge  and  skills 

•  Interest  in  SE  careers  generally  and  in  government  and  industry 

•  Knowledge  of  DoD  problem  areas  related  to  SE 

A  mix  of  methods  was  used  to  assess  the  success  of  the  SE  Capstone  courses  in  meeting  these 
objectives.  These  included  closed-  and  open-ended  questions  on  pre-  and  post-surveys  that  included 
questions  about  understanding  of  systems  engineering  and  career  awareness,  a  case  study,  and  weekly 
blog  assignments.  The  case  study,  administered  at  the  beginning  of  the  first  semester  and  again  at  the 
end  of  the  year,  was  designed  to  assess  students'  level  of  sophistication  in  applying  systems  engineering 
thinking  to  a  problem,  while  the  weekly  blogs  were  designed  to  provide  a  insights  into  how  the  students 
were  developing  specific  systems  engineering  competencies. 


2.1  STUDENT  SURVEYS 

Of  the  14  schools  that  received  RT-19  funding,  a  total  of  301  students  from  13  schools  responded  to  the 
RT-19  pre-survey  that  was  designed  to  assess  student  understanding  of  systems  engineering,  awareness 
of  systems  engineering  careers,  and  interest  in  those  careers.  Students  from  12  of  those  13  schools 
completed  the  pre-survey  in  September  2010;  students  at  one  school  (UMD)  completed  it  in  March 
2011. 

A  total  of  123  students  from  12  schools  completed  the  post-survey,  although  not  all  at  the  same  time 
and  not  all  to  the  same  survey.  Students  from  10  schools  responded  to  the  full  post-survey  between 
May  and  June  2011.  Students  from  Penn  State  University,  who  completed  a  one-semester  course  in  the 
fall,  responded  to  a  post-survey  that  did  not  have  two  questions  that  were  added  subsequently. 
Students  from  UMD  responded  to  the  pre-survey  twice,  with  the  results  that  they  did  not  respond  to 
questions  that  were  only  on  the  post-survey.  Finally,  there  were  no  responses  from  NPS,  which  was  not 
due  to  complete  the  course  until  fall  2011,  nor  from  Wayne  State,  which  did  not  respond  to  multiple 
prompts  from  the  research  team  regarding  completion  of  the  mandatory,  common  assessments. 

Overall,  the  total  post-survey  responses  (n=123)  were  41%  of  the  original  pre-surveys  (n=301)  and  the 
matched  surveys  (n=93)  were  31%.  However,  since  NPS  was  not  expected  to  finish  until  Fall  2011,  they 
should  not  be  considered  as  part  of  the  population.  If  NPS  is  removed,  the  total  post-survey  responses 
were  50%  of  the  pre-survey  responses  and  the  number  of  matched  pre-post  survey  responses  was  76% 
of  the  post-survey  responses. 

The  reduction,  and  lack  of  a  larger  match,  was  in  part  due  to  the  fact  that  some  students  who  took  the 
first-semester  course  did  not  go  on  to  the  second  semester,  while  at  a  few  schools,  new  students  joined. 
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For  example,  SMU  had  students  in  three  fall  semester  capstone  courses  with  75  students  total.  Although 
43  responded  to  the  pre-survey,  only  11  worked  on  DoD  projects  (according  to  the  Pi's  report)  and  went 
on  to  the  spring  semester.  At  Auburn,  only  15  of  the  original  33  went  on  to  the  second  semester. 

The  table  below  compares  the  number  of  students  in  each  semester,  as  reported  by  the  Pis  in  their 
reports,  the  number  of  surveys  received,  and  the  number  that  could  be  matched  pre-to-post: 


#  Students 

Semester  1 

#  Pre-survey 

responses 

#  Students 

Semester  2 

#  Post-survey 

responses 

#  Matched 

Responses 

AFA 

7 

10 

7 

6 

6 

AFIT 

5 

3 

5 

(1  student 
from  first 
semester  left; 

1  student 
joined) 

3 

3 

Auburn 

33 

41 

17 

(15  were  from 
first  semester) 

14 

7 

CGA 

20 

14 

24 

(4  joined  in 
spring) 

17 

11 

MUST 

30 

20 

30 

17 

13 

NPS 

38 

57 

38 

0  (to  come  fall 
2011) 

0 

PSU 

17 

17 

17 

15 

13 

SMU 

75 

43 

11 

7 

2 

Stevens 

24 

24 

19 

(5  graduate 
students  left  in 
spring) 

9 

9 

UMD 

15 

35 

(March  2011) 

37 

14 

12 

UVA 

17 

17 

16 

14 

13 

Wayne 

29 

16 

16  (all  new)* 

0 

0 

USMA 

4 

4 

4 

4 

4 
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USNA 

16 

0 

16 

3 

0 

Total 

330 

301 

257 

123 

93 

Total  without 

NPS 

292 

244 

219 

123 

93 

Table  5:  Administered  Surveys 


*  The  pre-surveys  were  collected  from  three  classes  of  students  in  the  fall  semester.  In  the  spring  semester,  one  of 
these  classes  was  repeated  with  a  wholly  new  set  of  students  who  were  not  asked  to  complete  the  pre-survey.  In 
addition,  as  noted  above,  no  Wayne  State  students  completed  the  post-survey. 


2.2  CASE  STUDY  ANALYSIS:  BRADLEY  LIGHTING  VEHICLE 

A  total  of  158  students  from  13  of  the  14  institutions  responded  to  the  RT-19  Bradley  Fighting  Vehicle 
pre-course  scenario  prompt  (the  exception  was  NPS)  and  97  students  from  12  institutions  responded  to 
the  post-scenario  prompt.  Wayne  State  and  NPS,  which  will  not  complete  the  capstone  course  until  fall 
2011,  did  not  respond  to  the  post-scenario  response. 

There  were  55  matched  responses.  Without  NPS,  this  is  35%  of  the  original  pre-course  case  study  group 
and  57%  of  post-course  case  study  group: 


#  Students 

Semester  1 

#  Pre-scenario 

responses 

#  Students 

Semester  2 

#  Post-scenario 

responses 

#  Matched 

Responses 

AFA 

7 

8 

7 

6 

3 

AFIT 

5 

3 

5 

1 

1 

Auburn 

33 

31 

17 

13 

6 

CGA 

20 

2 

24 

13 

1 

MUST 

30 

1 

30 

6 

0 

NPS 

38 

0 

38 

0 

0 

PSU 

17 

17 

17 

15 

15 

SMU 

75 

10 

11 

7 

6 

Stevens 

24 

21 

19 

8 

6 

UMD 

15 

32 

37 

10 

7 

UVA 

17 

16 

16 

13 

5 

Wayne 

29 

10 

16 

0 

0 
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USMA 

4 

4 

4 

4 

4 

USNA 

16 

3 

16 

1 

1 

Total 

330 

158 

257 

97 

55 

Total 

without 

NPS 

292 

158 

219 

97 

55 

Table  6:  Administered  BFV  Case  Study 


In  this  report,  two  data  sets  -  all  post-survey  responses  and  the  smaller  set  of  matched  pre-post  survey 
responses  -  were  analyzed  depending  on  the  survey  question.1  Thus  the  entire  set  of  responses  was 
used  for  those  questions  that  were  only  on  the  post-survey;  for  all  other  questions,  the  matched  set  was 
used. 

Demographics  of  the  Matched  Set  of  Students. 

Only  26  of  the  matched  pre-post  survey  population  (n=93)  were  Systems  Engineering  majors.  The 
majority  (n=67)  were  other  primarily  engineering  disciplines,  including  Mechanical,  Electrical,  Industrial, 
Software,  and  other  engineering  disciplines.  Students  in  Systems  Engineering  and  Mechanical 
Engineering  majors,  however,  formed  the  highest  concentration  of  majors  (n=26). 


Number 

Percent 

Mechanical  Engineering 

26 

27.9 

Systems  Engineering 

26 

27.9 

Electrical  Engineering 

16 

17.2 

Engineering  Management 

6 

6.5 

Software  Engineering 

4 

4.3 

Aeronautical  Engineering 

3 

3.2 

Computer  Science 

3 

3.2 

Industrial  Engineering 

3 

3.2 

Biomedical  Engineering 

1 

1.1 

Chemical  Engineering 

1 

1.1 

1  All  pre-survey  responses  were  analyzed  in  the  Interim  Report. 
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Civil  Engineering 

1 

1.1 

Computer  Engineering 

1 

1.1 

Operations  Research 

1 

1.1 

Product  Architecture 

1 

1.1 

Total 

93 

100.0 

Table  7:  Breakdown  of  Student  Majors  in  Matched  Population 

The  breakdown  by  institution  is  as  follows: 


Undergraduate 

SE  majors 

Undergraduates  SE 
major  +  another 
major 

Undergraduate 
majors/  Other 

Graduate  SE 
majors 

Graduate 

majors:/Other 

AFA 

1 

1 

[double  major  in 

Computer 

Engineering] 

4 

[2  Mechanical 
Engineering 

2  Electrical 
Engineering] 

0 

0 

AFIT 

0 

0 

0 

2 

1 

[Aeronautical 

Engineering] 

Auburn 

0 

0 

2 

[1  Computer 
Science 

1  Software 
Engineering] 

0 

5 

[2  Computer 

Science 

3  Software 
Engineering] 

CGA 

0 

0 

11 

[Mechanical 

Engineering] 

0 

0 

Military 

Academy 

1 

1 

[double  major  in 

Engineering 

Management] 

2 

[1  Engineering 
Management 

1  Operations 
Research] 

0 

0 

MUST 

0 

0 

3 

[Mechanical 

Engineering] 

10 

0 

PSU 

0 

0 

13 

[1  Engineering 

0 

0 
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Management 

1  Chemical 
Engineering 

3  Industrial 
Engineering 

4  Mechanical 
Engineering 

4  Electrical 
Engineering] 

SMU 

0 

0 

2 

[2  Electrical 
Engineering  (la 
double  major  in 
Marketing  & 

Math)] 

0 

0 

Stevens 

0 

2 

[1  undergrad 
receiving  SE 
certificate  while 
majoring  in 

Electrical 
Engineering;  1 
undergraduate 
receiving  SE 
graduate 
certificate  while 
majoring  in 
Mechanical 
Engineering] 

6 

[1  Mechanical 
Engineering 

1  Civil  Engineering 

4  Engineering 
Management] 

0 

1 

[Product 

Architecture] 

UMD 

0 

0 

12 

[2  Mechanical 
Engineering 

8  Electrical 
Engineering 

2  Aeronautical 
Engineering] 

0 

0 

UVA 

8 

0 

5 

[1  Biomedical 
Engineering 

1  Computer 
Engineering 

3  Mechanical 
Engineering] 

0 

0 

Contract  Number:  H98230-08-D-01 71 ,  D02,  TT02,  RT#01 9  Phase  III 

Report  No.  SERC-2011-TR-020 
October  31,  2011 


UNCLASSIFIED 


24 


UNCLASSIFIED 

Table  8:  Graduate  and  Undergraduate  Systems  Engineering  and  Other  Engineering  Majors  by  Institution 

The  26  systems  engineering  majors  (including  double  majors)  were  nearly  evenly  split  between  graduate 
students  and  undergraduates,  but  almost  all  of  the  non-systems  engineering  majors  were 
undergraduates,  as  were  almost  all  of  those  without  systems  engineering  experience  (either  through 
courses,  internships,  or  work): 


Graduates 

(n=14) 

Undergraduates 

(n=49) 

Systems  engineering  majors 

12 

14 

Other  majors 

7 

60 

With  systems  engineering  experience 

11 

23 

Without  systems  engineering  experience 

3 

26 

Table  9:  Graduate  Vs  Undergraduate-SE  and  Non  SE  Majors 


Finally,  it  should  be  noted  that  the  n's  for  each  question  analyzed  may  vary  even  within  the  matched  set 
if  a  student  did  not  answer  a  question  or  answered  a  question  on  the  pre-survey  but  not  on  the  post, 
and  vice  versa. 


2.3  UNDERSTANDING  OF  SYSTEMS  ENGINEERING 

Systems  engineering  is  a  complex  field  that  includes  interdisciplinary  approaches  to  design  and 
problem  solving;  a  corpus  of  diverse  theoretical  and  practical  models,  and  their  pedagogical 
applications;  and  program  offerings  that  vary  institutionally  at  the  undergraduate  and  graduate  levels 
in  curriculum  design,  course  requirements  and  overall  educational  objectives  (Brown  &  Scherer,  2000). 

On  both  the  pre-  and  post-surveys,  students  were  asked  to  respond  to  the  open-ended  question,  "What 
is  systems  engineering?  Define  it  as  best  you  can."  Concise  professional,  technical,  and  academic 
definitions  that  summarized  the  field  in  terms  of  methodology,  process,  and  discipline  were  found  in 
documents  created  by  military,  aerospace,  technical  and  professional  organizations,  including  the 
Defense  Acquisition  University  (DAU),  the  National  Aeronautics  and  Space  Administration  (NASA  NPR 
7123. 1A;  NASA  Systems  Engineering  Processes  and  Requirements),  the  International  Council  on  Systems 
Engineering  (INCOSE),  the  Institute  of  Electrical  and  Electronics  Engineers  (IEEE),  and  the  Engineering 
Industries  Association  (EIA),  and  also  in  textbooks  on  systems  engineering.  Below  are  two  examples,  one 
from  the  DAU  and  one  from  NASA  NPR: 

"For  DoD,  systems  engineering  is  the  set  of  overarching  processes  that  a  program  team  applies 
to  develop  an  operationally  effective  and  suitable  system  from  a  stated  capability  need. 

Systems  engineering  processes  apply  across  the  acquisition  life  cycle  (adapted  to  each  phase) 
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and  serve  as  a  mechanism  for  integrating  capability  needs,  design  considerations,  design 
constraints,  and  risk,  as  well  as  limitations  imposed  by  technology,  budget,  and  schedule.  The 
systems  engineering  processes  should  be  applied  during  concept  definition  and  then 
continuously  throughout  the  life  cycle.... 

Systems  engineering  is  a  broad  topic  that  includes  hardware,  software,  and  human  systems.  It  is 
an  interdisciplinary  approach  for  a  structured,  disciplined,  and  documented  technical  effort  to 
simultaneously  design  and  develop  systems  products  and  processes  for  creating  and  integrating 
systems  (hardware,  software,  and  human)  to  satisfy  the  operational  needs  of  the  customer.  It 
transforms  needed  operational  capabilities  into  an  integrated  system  design  through  concurrent 
consideration  of  all  life  cycle  needs.  As  systems  become  larger  and  more  complex,  the  design, 
development,  and  production  of  such  systems  or  system  of  systems  (SoS)  require  the 
integration  of  numerous  activities  and  processes.  Systems  engineering  is  the  approach  to 
coordinating  and  integrating  all  these  acquisition  life  cycle  activities.  It  integrates  diverse 
technical  management  processes  to  achieve  an  integrated  systems  design"  (Defense  Acquisition 
University,  2011,  p.  167). 

"Systems  engineering  at  NASA  requires  the  application  of  a  systematic,  disciplined  engineering 
approach  that  is  quantifiable,  recursive,  iterative,  and  repeatable  for  the  development, 
operation,  maintenance,  and  disposal  of  systems  integrated  into  a  whole  throughout  the  life 
cycle  of  a  project  or  program.  The  emphasis  of  systems  engineering  is  on  safely  achieving 
stakeholder  functional,  physical,  and  operational  performance  requirements  in  the  intended  use 
environments  over  the  system's  planned  life  within  cost  and  schedule  constraints" 

(National  Aeronautical  Space  Administration,  2007). 

Two  sets  of  frequently  occurring  keywords  were  extracted  from  the  definitions  in  these  documents.  The 
first  set  included  eighteen  non-technical  words  (including  compound  words)  that  relate  to  systems 
engineering;  the  second  included  20  technical  words  that  were  more  specific  to  systems  engineering 
(List  1:  generic  systems  engineering  base  words2) 


APPROACH 

CUSTOMER 

CROSS-DISCI  PLI  NARY 

DESIGN 

DEVELOPMENT 

DISCIPLINE 

EFFICIENT 

INTERDISCIPLINARY 

MANAGE 

MULTI-DISCIPLINARY 

PROBLEM  SOLVING 

PROJECT 

PRODUCT 

PROCESS 

REQUIREMENTS 

SYSTEM 

TEAM 

TEAMWORK 

Table  10:  List  1  -  Generic  Systems  Engineering  Base  Words 


BALANCE 

COLLABORATE 

COMPLEX 

COMPONENT 

COMMUNICATION 

CONCEPT  OF 
OPERATIONS3 

CYCLE 

DEFINITION 

DOCUMENTATION 

INTEGRATE 

LIFECYCLE 

NEEDS 

RISK 

SOLUTION 

STAKEHOLDER 

SUBSYSTEM 

TECHNICAL 

TECHNOLOGY 

VALIDATION 

VERIFICATION 

Table  11:  List  2  -  Systems  Engineering-Specific  Terms 


2  Cognate  and  plural  forms,  not  included  in  these  lists,  were  included  in  the  lists  used  in  the  analysis. 

3  AntWordProfiler,  the  concordance  and  text  comparison  tool  that  the  assessment  team  used,  did  not  allow  for  compound 
words  to  be  examined.  As  a  result,  terms  such  as  "concept  [of]  operations"  or  "life  cycle"  had  to  be  split  and  analyzed  as 
separate  words. 
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Using  a  free,  open-source  application  called  AntWordProfiler,4  lists  of  keywords  were  compared,  first 
using  the  expert  definitions  and  then  student  responses.  A  score  of  100  percent  for  both  lists  would 
have  included  every  single  keyword  but  would  not  have  been  a  readable  or  concise  definition.  One 
disadvantage  to  using  AntWordProfiler  was  that  it  scored  responses  higher  when  keywords  were 
repeated.  To  lessen  this  effect,  we  transformed  the  original  data  and  definitions  by  removing  duplicate 
terms  and  replacing  them  with  non-keyword  placeholders  in  order  to  avoid  higher  scores  based  on 
redundancy. 

None  of  the  expert  definitions  scored  over  32  percent  on  either  list.  Thus  when  the  expert  definitions 
were  entered  into  AntWordProfiler,  the  average  score  for  List  1  was  15.8  percent,  while  the  average 
score  for  List  2  was  19.96  percent.  With  one  exception  (INCOSE),  the  experts  had  almost  twice  as  many 
words  from  List  2  compared  to  List  1.  In  other  words,  the  experts  used  a  higher  percentage  of  the 
technical  words  than  the  generic  ones.  This  confirmed  that  the  two  lists  were  distinct  enough  to  make 
their  use  worthwhile. 


Definition  Source 

List  1 

List  2 

Air  Force  Academy  website 

16.0% 

32.0% 

Institute  of  Electrical  and  Electronics  Engineers  (IEEE) 

15.4% 

23.1% 

Engineering  Industries  Association  (EIA) 

17.1% 

22.9% 

International  Council  on  Systems  Engineering  (INCOSE)  1 

14.3% 

10.2% 

International  Council  on  Systems  Engineering  (INCOSE)  2 

18.6% 

11.6% 

Table  12:  Percent  of  Words  Used  in  Expert  Definitions 


In  order  to  understand  how  such  seemingly  low  percentages  can  apply  to  expert  definitions,  we  need  to 
remember  that  many  of  the  words  on  the  two  lists  are  likely  to  be  used  only  occasionally.  The  full 
definitions  that  were  used  are  below,  with  List  1  words  highlighted  in  red  and  List  2  words  in  highlighted 
in  green: 

AFA  definition: 

Systems  engineering  is  an  interdisciplinary  engineering  process  that  evolves,  verifies,  and  documents  an 
integrated,  life-cycle-balanced  set  of  systems  solutions  that  satisfy  customer  needs. 


4  AntWordProfiler  1.200m  was  developed  by  Dr.  Laurence  Anthony,  an  English  language  professor  at  the  Center  for  English 
Language  Education  in  Science  and  Engineering  (CELESE),  School  of  Science  and  Engineering,  Waseda  University  (Japan). 
According  to  Anthony's  website,  AntWord  Profiler  is  "[a]  freeware  word-profiling  program  for  Windows  and  Macintosh  OS  X, 
similar  to  Paul  Nation's  RANGE  program."  The  RANGE  program  was  originally  developed  by  Nation  to  examine  lists  of  common 
words  and  perform  a  concordance  between  various  lists  and  journalistic  and  more  literary  texts  in  order  to  examine  the 
number  and  types  of  words  needed  for  reading  and  writing  certain  vernaculars  (Nation  I.S.P.,  2004)  (Nation,  I.S.P.,  2006).  It  was 
modified  and  re-envisioned  by  Anthony  to  meet  his  own  research  interests  in  corpus  linguistics,  genre  analysis,  and  natural 
language  processing. 
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IEEE  definition: 

Systems  engineering  is  an  interdisciplinary,  collaborative  approach  that  derives,  evolves  and  verifies  a 
life-cycle  balanced  system  solution  which  satisfies  customer  expectations  and  meets  public 
acceptability. 

EIA  definition: 

Systems  engineering  is  an  interdisciplinary  approach  that  encompasses  the  entire  technical  effort,  and 
evolves  into  and  verifies  an  integrated  and  life  cycle  balanced  set  of  system  people,  products  and 
process  solutions  that  satisfy  customer  needs. 

INCOSE  definition  1: 

Systems  engineering  integrates  all  the  disciplines  and  specialty  groups  into  a  team  effort  forming  a 
structured  development  process  that  proceeds  from  concept  to  operation.  Systems  engineering 
considers  both  the  business  and  technical  needs  of  all  customers  with  the  goal  of  providing  a  quality 
product  that  meets  the  user  needs. 

INCOSE  definition  2: 

Systems  engineering  is  an  interdisciplinary  approach  and  means  to  enable  realization  of  successful 
systems.  It  focuses  on  defining  customer  needs  and  required  functionality  early  in  the  development 
cycle,  documenting  requirements,  then  proceeding  with  design  synthesis  and  system  validation  while 
considering  the  complete  problem. 

We  expected  that  students  entering  Capstone  courses  without  prior  knowledge  of  systems  engineering 
would  initially  produce  responses  using  a  lower  percentage  of  both  sets  of  listed  words  than  the 
experts  and  that,  as  they  increased  in  their  systems  engineering  content  knowledge  and  practical 
experience  during  the  course  of  the  year,  they  would  continue  to  use  the  generic  terms  but  add  more 
of  the  technical  terms.  We  also  predicted  that  they  would  shift  away  from  an  understanding  of  systems 
engineering  as  a  form  of  project  management  toward  a  more  holistic  understanding  of  systems 
engineering  that  included  knowledge  of  systems,  subsystems,  life  cycle  models,  etc. 

Results 

For  List  1,  the  student  percentage  was  only  a  few  percentages  points  below  the  expert  percentage  in 
both  the  pre-  and  post-survey.  For  List  2,  in  contrast,  it  was  substantially  below  even  the  INCOSE 
definition  (11.6%).  For  both  lists,  the  student  averages  increased  from  pre-  to  post-survey  for  the  84 
matched  students— those  who  responded  to  the  question  on  both  the  pre-  and  post-surveys— although 
the  changes  were  not  statistically  significant  for  either  list: 


List  1  Pre-Survey 

12.35% 

List  2  Pre-Survey 

4.09% 

List  1  Post-Survey 

13.80% 

List  2  Post-Survey 

5.65% 

Table  13:  List  1  and  List  2  Averages  in  Percent:  All  Students  (n=84) 
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When  we  look  at  the  subgroup  with  the  least  to  learn— those  who  reported  that  they  had  prior  systems 
engineering  experience  (including  coursework)— we  see  that  they  began  with  slightly  higher  scores  on 
List  1  and  considerably  higher  scores  on  List  2.  As  might  be  expected,  they  did  not  improve  very  much: 


List  1  Pre-Survey 

12.83% 

List  2  Pre-Survey 

5.16% 

List  1  Post-Survey 

13.72% 

List  2  Post-Survey 

5.43% 

Table  14:  List  1  and  List  2  Averages  in  Percent:  Prior  SE  Experience  (n=41) 


However,  when  we  look  at  the  two  subgroups  with  the  most  to  learn— those  who  reported  that  they 
had  no  prior  systems  engineering  experience  (including  coursework)  and  undergraduates— we  see  lower 
percentages  at  start  and  larger  changes  for  both  groups.  We  also  see  that  those  with  no  prior  systems 
engineering  experience  not  only  gained  more,  but  also  ended  with  a  slightly  higher  percentage  gains 
than  the  entire  group,  indicating  that  SE  Capstone  courses  had  a  positive  impact  on  student  learning  of 
the  terminology  related  to  SE  methodology,  process,  and  discipline: 


List  1  P  re -Survey 

11.50% 

List  2  Pre-Survey 

2.96% 

List  1  Post-Survey 

14.33% 

List  2  Post-Survey 

5.93% 

Table  15:  List  1  and  List  2  Averages  in  Percent:  No  Prior  SE  Experience  (n=41) 


List  1  Pre-Survey 

11.88% 

List  2  Pre-Survey 

3.40% 

List  1  Post -Survey 

13.23% 

List  2  Post-Survey 

5.50% 

Table  16:  List  1  and  List  2  Averages  in  Percent:  Undergraduates  (n=67) 


Although  the  change  was  not  statistically  significant  for  either  subgroup  for  List  1,  it  was  significant  for 
both  subgroups  for  List  2.  For  students  who  did  not  have  prior  systems  engineering  experience,  the  List 
2  post-survey  scores  increased  significantly  at  the  p  =.00  level ,  f(40)  =  2.82,  p  <.01,  p  =  .007,  d  =  0.89. 
The  strength  of  the  relationship  was  >.5,  a  high  categorization  of  strength  (Cohen,  1988). 

For  undergraduates,  List  2  scores  also  increased  significantly  at  the  p  <.01  level,  f(66)  =  2.64,  p  <.01,  d  = 
0.65.  The  strength  of  the  relationship  was  >  .5,  above  medium  categorization  of  strength  (Cohen,  1988). 

In  other  words,  the  greatest  change  was  in  the  number  of  specifically  systems  engineering  words  (List  2) 
used  by  those  with  the  least  prior  knowledge  of  systems  engineering. 

Case  Study  Analysis:  The  Bradley  Fighting  Vehicle 

As  a  second,  more  holistic,  way  of  assessing  student  understanding  of  systems  engineering,  students 
were  asked  to  read  and  respond  to  a  case  study  based  on  the  well-known  history  of  the  U.S.  Army's 
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development  of  the  Bradley  Fighting  Vehicle  (BFV).  In  many  disciplines,  case  studies  are  used  to 
generate  classroom  discussion,  to  help  students  develop  their  problem-solving  skills  and  to  force  them 
to  articulate  the  criteria  they  use  when  analyzing  problem  scenarios  (Heywood,  2005).  The  case  study 
was  thus  another  way  to  evaluate  the  RT-19  students  across  universities,  one  that  went  beyond  multiple 
choice  questions  or  self-assessments. 

The  brief  case  study  that  was  developed  was  based  on  research  into  the  history  and  development  of  the 
Bradley  Fighting  Vehicle;  a  brief  case  study  of  the  Vasa,  another  complex  defense  system  in  its  time;  and 
consultation  with  systems  engineering  professors  at  Stevens  Institute  of  Technology.  It  included 
historical  background  information  that  described  the  vehicle's  changing  requirements;  conflicts  over 
safety  and  live  fire  testing;  concerns  such  as  documentation,  time,  and  budget  constraints;  and 
professional  ethics  and  communication.  Film  clips  from  The  Pentagon  Wars,  a  satirical  portrait  of  the 
BFV's  prolonged  and  embattled  development  process,  were  interspersed  throughout  the  text  as 
instructional  anchors  to  make  the  assessment  more  engaging  to  students. 

After  reading  the  scenario  and  watching  the  clips,  students  were  asked  to  respond  to  the  following 
prompt: 


"Could  the  problems  encountered  in  developing  the  Bradley  Fighting  Vehicle  have  been 
avoided?  Explain  your  answer." 

A  slightly  different  prompt  was  used  for  the  post-course  case  study  analysis: 

"Now  that  you  have  completed  a  systems  engineering  project,  do  you  think  that  the  problems 
encountered  in  developing  the  Bradley  Fighting  Vehicle  project  could  have  been  avoided? 
Explain  your  answer." 

It  was  expected  that  prior  to  the  course,  the  students  would  use  mostly  imprecise  or  generic  terms  to 
describe  the  problems  encountered  by  the  BFV  developers  and  would  have  only  vague  ideas  as  to  how 
to  address  them.  After  the  course,  in  contrast,  it  was  anticipated  that  they  would  incorporate  more 
technical  terminology  and  use  more  specific  systems  engineering  concepts.  For  instance,  if  at  first  the 
students  used  general  terms  like  "time,"  "money,"  or  "politics,"  they  could  be  expected  to  use  more 
technical  terms,  such  as  "competing  requirements,"  "stakeholder  interests,"  "life-cycle  models,"  etc.,  in 
their  post-analysis. 

Results 

Given  the  open-ended  nature  of  the  prompt,  the  students'  varied  experience  with  systems  engineering, 
and  the  diversity  of  their  educational  and  disciplinary  backgrounds,  it  is  not  surprising  that  there  was  a 
wide  range  of  understanding  exhibited  in  the  first  set  of  responses  to  the  case  study.  For  example,  there 
were  many  responses  that  included  only  very  vague  descriptions  of  the  BFV  and  inconclusive 
recommendations  for  the  future,  such  as  this: 

"Problems  could  have  been  avoided  if  requirements  were  carefully  followed  and  did  not  change  so 
frequently.  By  keeping  in  mind  what  [the]  vehicle  was  originally  meant  to  do,  the  addition  of 
extraneous  features  could  have  been  avoided." 
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On  the  other  hand,  very  few  students  had  sophisticated  analyses  of  the  problems  and  recommendations 
for  the  future,  using  systems  engineering  terminology  and  concepts  such  as  "product  life  cycle,"  "needs 
statement,"  "spiral  development  approach,"  etc.  Here  is  one  example  of  a  more  sophisticated  response 
from  the  pre-case  study  analysis: 

"Most  of  the  issues  addressed  in  the  scenario  could  have  been  avoided  with  inclusion  of  some  now 
common  systems  engineering  practices  and  just  a  general  focus  on  requirements  validation. 
Requirements  creep  was  the  primary  reason  for  most  of  the  problems  observed  in  the  Bradley 
program.  A  PEO  capable  of  maintaining  a  design  baseline  would  have  made  a  dramatic  impact.  As 
the  program  dragged  on,  new  requirements  were  levied  on  the  contractor  and  the  design  kept 
diverging  from  the  original  requirements.  Although  this  is  common,  a  spiral  development  approach 
could  have  accelerated  the  program  and  got  a  baseline  configuration  to  the  users.  This  would  have 
resulted  in  a  more  focused  test  program  because  incremental  testing  would  have  identified 
problems  earlier  and  it  would  have  forced  design  and  testing  personnel  to  work  towards  a 
solution." 

An  analytic  rubric  was  developed  from  the  first  set  of  student  responses,  with  the  goal  of  measuring 
increasing  complexity  of  systems  engineering  knowledge.5  A  life  cycle  model  provided  a  guide  (Sage, 
2000).  Simpler  responses  were  classified  as  addressing  primarily  the  "Definition"  phases,  while  more 
complex  responses  extended  to  the  "Development"  and  "Deployment"  phases: 


5  While  rubrics  are  often  developed  before  any  documents  are  received— for  example,  to  guide  students  in  assignments-they 
can  also  be  created  a  posteriori,  after  the  data  has  been  examined  for  patterns,  as  was  the  case  here  (Leydens,  Moskval,  & 
Pavelich,  2004). 
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Figure  5:  SE  Life  Cycle  Model 

The  scoring  rubric  was  divided  in  three  categories: 

A.  Demonstration  of  understanding  of  systems  engineering  as  an  approach  to  problem  solving: 

Identifies  one  or  more  central  problems  of  the  BFV  using  appropriate  Systems  Engineering 
vocabulary 

B.  Demonstration  of  understanding  of  systems  engineering  as  a  process  involving  complex 
systems:  Provides  recommendations  and  solutions  to  the  problems  mentioned  in  Category  A, 
ideally  moving  beyond  the  requirements  definition  stage  to  later  phases  of  development 

C.  Demonstration  of  understanding  of  professional  traits  in  the  Systems  Engineering  discipline: 

Any  statement  that  implicitly  or  explicitly  argues  for  the  importance  of  communication,  working 
in  a  team,  and  attending  to  ethical  considerations  when  defining,  designing,  or  developing  a 
complex  system. 


Key  competencies  were  selected  from  the  SPRDE-SE/PSE  Competency  Model  and  mapped  onto  each 
category.  The  assumption  was  that  when  students  identified  some  of  the  central  problems  of  the  BFV 
(e.g.,  requirements  creep,  failure  to  meet  original  product  specifications,  inattention  to  budget  limits 
and  safety  issues,  inattention  to  product  life  cycle,  poor  implementation  of  risk  management  plans,  and 
miscommunication  or  lack  of  communication  among  different  interests  across  various  stages  of  the 
BFV's  design),  they  also  demonstrated  understanding  of  select  systems  engineering  competencies. 
These  competencies  also  addressed  potential  solutions  that  could  be  provided  to  ameliorate  or 
troubleshoot  problems  that  ranged  from  the  technical  to  the  political. 

The  competencies  chosen  were  the  following: 
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Units  of  competency  mapped  to  Category  A: 

#4  Stakeholder  requirements  definition  -  Element  4.  Work  with  the  user  to  establish  and  refine 
operational  needs,  attributes,  performance  parameters,  and  constraints  that  flow  from  the  Joint 
Capability  Integration  and  Development  System  described  capabilities,  and  ensure  all  relevant 
requirements  and  design  considerations  are  addressed. 

#5  Requirements  analysis  -  Element  5.  Ensure  the  requirements  derived  from  the  customer-designated 
capabilities  are  analyzed,  decomposed,  functionally  detailed  across  the  entire  system,  feasible  and 
effective. 

#18  Requirements  management  -  Element  23.  Use  Requirements  Management  to  trace  back  to  user- 
defined  capabilities  and  other  sources  of  requirements,  and  to  document  all  changes  and  the  rationale 
for  those  changes. 

#27  Problem  solving  -  Element  43.  Make  recommendations  using  technical  knowledge  and  experience, 
developing  a  clear  understanding  of  the  system,  identifying  and  analyzing  problems  using  a  Total 
Systems  approach,  weighing  the  relevance  and  accuracy  of  information,  accounting  for 
interdependencies,  and  evaluating  alternative  solutions. 

Units  of  competency  mapped  to  Category  B: 

#8  Integration  -  Element  11.  Manage  the  technical  issues  that  arise  as  a  result  of  the  integration 
processes  that  feed  back  into  the  design  solution  process  for  the  refinement  of  the  design. 

#9  Verification  -  Element  13.  Verify  the  system  elements  against  their  defined  requirements  (build-to 
specifications). 

#10  Validation  -  Element  14.  Evaluate  the  requirements,  functional  and  physical  architectures,  and  the 
implementation  to  determine  the  right  solution  for  the  problem 

#19  Risk  management  -  Element  24.  Create  and  implement  a  Risk  Management  Plan  encompassing  risk 
identification,  analysis,  mitigation  planning,  mitigation  plan  implementation,  and  tracking  throughout 
the  total  life-cycle  of  the  program.  Element  25.  Apply  risk  management  at  the  earliest  stages  of  program 
planning  and  continue  throughout  the  total  life  cycle  of  the  program  through  the  identification  of  risk 
drivers,  dependencies,  root  causes,  and  consequence  management. 

#25  -  System  of  systems  -  Element  41.  Oversee  the  planning,  analyzing,  organizing,  and  integrating  the 
capabilities  of  a  mix  of  existing  and  new  systems  into  a  SoS  capability  greater  than  the  sum  of  the 
capabilities  of  the  constituent  parts. 

Units  of  competency  mapped  to  Category  C: 

#26  Communication  -  Element  42.  Communicate  technical  and  complex  concepts  in  a  clear  and 
organized  manner,  both  verbally  and  in  writing,  to  inform  and  persuade  others  to  adopt  and  act  on 
specific  ideas. 
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#28  Strategic  thinking  -  Element  43.  Formulate  and  ensure  the  fulfillment  of  objectives,  priorities,  and 
plans  consistent  with  the  long-term  business  and  competitive  interests  of  the  organization  in  a  global 
environment 

#29  Professional  ethics  -  Element  56.  Maintain  strict  compliance  to  governing  ethics  and  standards  of 
conduct  in  engineering  and  business  practices  to  ensure  integrity  across  the  acquisition  life-cycle 

#24  Leadership  -  Element  40.  Lead  teams  by  providing  proactive  and  technical  direction  and  motivation 
to  ensure  the  proper  application  of  systems  engineering  processes  and  the  overall  success  of  the 
technical  management  process. 

Two  separate  raters  used  a  grading  rubric  comprised  of  the  three  categories  described  above  (A,  B,  C), 
with  each  of  the  categories  graded  on  a  scale  of  1  to  5  points.  The  lowest  possible  score  was  therefore 
three  points  (1  in  each  category)  and  the  highest  was  15. 

Low  answers  (3-4),  as  demonstrated  in  the  student  example  below,  did  not  identify  problems  or  make 
recommendations  using  appropriate  systems  engineering  concepts  and  terminology.  The  response 
below  scored  a  4  on  the  rubric  because  it  used  vague  language  to  identify  problems  with  the  vehicle's 
design  (Category  A,  2  points);  it  provided  no  concrete  recommendations  to  solving  some  of  the 
problems  encountered  (Category  B,  1  point);  and  it  also  failed  to  mention  any  professional  traits  in  the 
systems  engineering  discipline  (Category  C,  1  point): 

"The  issue  was  a  lack  of  aim  for  the  vehicle.  The  goal  of  the  project  was  changed  so  many  times 
over,  that  by  the  end  of  the  project,  it  did  not  even  meet  its  original  goals.  If  there  had  been  a 
clear  aim  from  the  beginning,  one  that  was  not  subject  to  so  much  change,  the  Bradley  Fighting 
Vehicle  may  have  just  been  a  troop  transport  as  intended." 

Medium  Low  responses  (5-7)  satisfactorily  used  systems  engineering  language  and  concepts  to  identify 
problems  and  make  recommendations  related  to  the  BFV's  design  and  development  process.  In  the 
student  example  below,  the  response  was  given  a  score  of  7  because  it  identified  the  main  problem  as 
competing  requirements  (Category  A,  3  points)  and  provided  some  recommendations  to  solving  a  cited 
problem  (Category  B,  3  points).  However,  no  professional  traits  of  systems  engineering  (Category  C,  1 
point)  were  discussed: 

"Yes,  problems  in  the  Bradley  vehicle  definitely  could  have  been  avoided.  After  going  through 
the  systems  engineering  course,  I  have  learned  that  a  large  majority  of  your  costs  are  committed 
once  you  begin  designing  your  system.  I  learned  that  you  actually  commit  to  costs  much  ahead 
of  actually  spending  the  money.  That’s  why  it  is  important  to  make  sure  your  original  design 
meets  all  of  the  necessary  requirements.  With  the  Bradley,  many  of  the  requirements  kept 
changing.  This  led  to  changes  in  the  design  and  causing  the  cost  of  the  project  to  reach 
astonishing  levels.  The  Bradley  could  have  been  designed  more  effectively  and  much  cheaper  if 
the  design  requirements  were  constant." 

Medium  High  responses  (8-11)  often  provided  a  more  holistic  perspective  of  various  problems 
experienced  throughout  the  vehicle's  life  cycle,  from  the  initial  requirements  phase  to  later  verification 
and  validation  phases.  The  response  below  was  given  a  score  of  11  (Categories  A  and  B,  5  points  each, 
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Category  C,  1  point)  for  citing  complex  systems  engineering  concepts  such  as  iterative  testing,  systems 
integration,  and  trade-off  analysis  as  important  factors  in  improving  the  design  process.  However,  this 
student  did  not  discuss  professional  traits  of  systems  engineering,  such  as  improving  communication 
with  the  design  team  or  with  stakeholders: 

"I  think  if  they  approached  the  Bradley  project  with  a  better  iterative  approach  many  of  the 
problems  could  have  been  avoided.  Each  time  a  new  design  was  developed,  there  was  no  design 
or  cost  trade-off  analysis  which  would  have  helped  evaluate  what  is  important  in  the  design.  Per 
usual  in  systems  design  problems,  the  client  did  not  know  what  they  wanted  and  this  led  to 
multiple  models  being  redesigned.  If  the  Army  had  used  smaller  iterations  in  their  designs,  they 
could  have  expressed  these  changes  to  the  client  without  getting  so  in  depth  with  each  model. 
There  was  also  a  lot  of  trouble  resulting  from  poor  systems  testing.  They  chose  to  only  test 
individual  components  and  furthermore,  to  only  do  computer  model  simulation.  This  is  a  bad 
decision.  If  they  had  done  real,  fully  integrated  systems  testing,  they  could  have  ensured  the 
system  will  work  in  real  life  situations  to  maintain  the  safety  of  the  troops  using  them.  They  also 
should  have  done  user  testing  to  get  people  who  are  unfamiliar  with  the  exact  design  to  test 
human  factors.  These  testing  techniques  would  have  prevented  conflicts  among  officers  and  the 
trials  could  have  been  avoided,  saving  both  time  and  money. 

Finally,  high  responses  (12-15),  although  infrequent,  diagnosed  design  problems  and  provided  high-level 
recommendations  using  appropriate  systems  engineering  terminology.  Most  important,  unlike  the  lower 
responses,  they  were  much  more  likely  to  reference  the  importance  of  communication,  leadership,  and 
aspects  of  the  stakeholder  and  team  relationship  as  integral  to  effective  delivery  and  product  function. 
The  student  response  below  received  a  high  score  of  13,  with  4  points  in  Category  A,  4  points  in 
Category  B,  and  5  points  in  Category  C.  While  the  "Medium  High"  response  above  did  better  in 
Categories  A  and  B  by  referencing  multiple  systems  engineering  concepts  and  phases  of  development, 
the  response  below  performed  better  in  Category  C,  and  received  5  points  because  the  student 
addressed  issues  of  communication: 

"I  think  now  after  having  completed  a  Systems  Engineering  project,  it  is  easy  to  recognize  the 
importance  of  several  competencies  that  we  used  over  the  span  of  the  project.  Probably  the 
most  relevant  competency  to  our  project,  and  the  Bradley  Fighting  Vehicle  project  is  the 
concept  of  risk  management.  If  the  team  in  the  BFV  project  would  have  had  the  foresight  to  use 
risk  management,  and  alter  the  design  when  the  risks  became  too  high,  a  lot  of  time  and  money 
could  have  been  saved.  We  had  a  similar  experience  during  our  semester,  when  one  of  the 
components  in  our  project  wasn't  as  useful  as  we  had  originally  intended.  To  mitigate  the  risk, 
we  drastically  changed  our  design,  which  in  turn  saved  us  a  lot  of  time  continuing  down  a 
difficult  path.  The  design  change  ended  up  saving  our  project  and  allowing  us  to  complete  our 
goals  within  the  given  time  constraints.  The  next  competency  that  was  the  most  useful  during 
the  semester  is  the  notion  of  communication.  This  competency  also  applies  to  the  BFV  project, 
because  if  strong  communication  was  used  between  the  team,  and  the  customers  that  were 
funding  the  project,  then  different  results  could  have  been  achieved.  The  most  important  task 
during  communication  is  the  requirements  gathering  process.  It  is  necessary  that  both  the  team 
working  on  the  project,  and  the  customers  funding  the  project  are  on  the  same  page,  and  that 
the  requirements  are  feasible  given  other  constraints  of  the  project.  The  bottom  line  is  that  not 
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everything  can  be  accounted  for  when  performing  any  Systems  Engineering  project.  However,  if 
the  team  is  adequately  trained  in  the  various  competencies,  the  team  will  be  able  to  quickly 
adapt  and  mitigate  the  risks  that  arise  during  the  process  and  produce  a  product  that  still  meets 
the  requirements." 

Each  of  the  two  raters  first  scored  a  set  of  10  student  responses  for  both  the  pre-and-post  assessments. 
Inter-rater  reliability  was  checked  using  Cohen's  kappa.  For  Categories  A  and  B,  the  Kappa  statistics  of 
.43  and  .33  indicated  moderate  and  fair  inter-rater  reliability,  with  the  difference  the  result  of  one  rater 
consistently  scoring  higher  than  the  other. 6  In  addition,  the  intraclass  coefficient,  another  test  of 
consistency  using  a  one-way  repeated  measures  ANOVA  where  raters  (assumed  to  be  randomly 
selected)  were  examined  against  their  responses,  gave  significant  results  for  categories  A,  B,  and  C:  .71, 

p>.001. 

Results 

The  Pis  at  seven  schools  (Auburn,  UMD,  Naval  Academy,  SMU,  Stevens,  UVA,  Wayne  State)  assigned  the 
scenario  to  their  students  at  the  beginning  and  end  of  the  course,  asking  them  to  complete  it 
individually  outside  of  class.  There  were  a  total  of  55  matched  responses. 

Student  responses  changed,  although  not  dramatically,  from  pre-  to  post-course.  Below  is  one  example 
of  a  student  who  improved  from  an  initial  score  of  4— the  minimum  was  3— to  a  post-response  score  of 
8,  or  from  "Low"  to  "Medium  High": 

Pre-course  response: 

Yes.  The  issue  was  a  lack  of  aim  for  the  vehicle.  The  goal  of  the  project  was  changed  so  many 
times  over,  that  by  the  end  of  the  project,  it  did  not  even  meet  its  original  goals.  If  there  had  been 
a  clear  aim  from  the  beginning,  one  that  was  not  subject  to  so  much  change,  the  Bradley  Fighting 
Vehicle  may  have  just  been  a  troop  transport  as  intended. 

The  student  received  a  score  of  4  because  problems  were  identified  in  a  vague  manner  (Category  A,  2 
points;  Category  B,  1  point),  and  because  there  was  no  mention  of  any  systems  engineering  professional 
traits,  such  as  improved  communication  between  stakeholders  and  engineers  (Category  3,  1  point). 

Here  is  the  same  student's  post-course  response: 

Responding  to  this  prompt  at  the  end  of  the  year,  I  think  I  am  much  better  qualified  to  answer 
this  from  a  systems  engineering  standpoint.  Initially,  it  seemed  like  they  had  a  clear  purpose  for 
the  vehicle,  to  transport  troops  on  the  battle  field.  Still,  it  did  not  seem  like  they  really  designed 
this  vehicle  in  a  systematic  way.  The  first  step  would  have  been  to  identify  the  problem,  perhaps 
that  the  army  did  not  have  an  agile  troop  transport  for  the  battlefield.  The  next  step  would  have 
been  to  come  up  with  use  cases  for  such  a  transport  and  from  there  to  get  a  list  of 
requirements.  In  the  actual  situation,  there  were  constant  changes  that  altered  the  design,  that 
were  not  part  of  the  initial  requirements  and  that  changed  the  vehicle  entirely.  Perhaps,  some 


6  While  the  guidelines  for  Kappa  agreement  beyond  chance  have  been  debated,  .43  is  acceptable  for  most  purposes  (Banerji, 
Capozzoli,  McSweeney,  &  Sinha,  1999,  p.  6). 
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of  these  features  were  needed,  but  the  project  would  have  benefitted  greatly  from  trade  off 
analysis,  something  that  was  never  performed.  If  initially  the  vehicle  was  to  have  no  weapons  or 
armor,  and  carry  7  troops,  it  may  be  acceptable  that  certain  features  are  added  that  reduce  the 
capacity  to  carry  personnel,  but  that  is  a  decision  that  should  be  made  in  a  controlled  way,  not 
on  a  whim.  Overall,  the  vehicle  design  and  project  as  a  whole  was  a  failure  from  a  systems  stand 
point  and  it  is  clear  why  systems  engineering  is  a  growing  and  much  needed  field. 

This  response  demonstrated  an  ability  to  identify  problems  with  Bradley  Fighting  Vehicle's  design  and 
development  by  describing  the  impact  of  competing  requirements  (A,  3  points).  In  addition,  the  student 
provided  recommendations  from  a  systems  standpoint  using  systems  engineering  language,  as 
evidenced  by  the  references  to  tradeoff  analysis  and  problem  definition  that  moved  beyond  the 
requirements  phase  of  development  (B,  4  points).  While  there  was  no  explicit  reference  to  any 
professional  traits  of  systems  engineers,  such  as  communication  or  leadership  (C,  1  point),  overall  the 
student  demonstrated  improved  knowledge  of  systems  engineering  processes,  terminology,  and 
understanding  of  problem  areas  and  received  a  final  total  score  of  8,  compared  to  the  initial  score  of 
4.  While  a  total  score  of  8  is  only  on  the  midpoint  of  the  scoring  scale,  this  improvement  is  reasonable 
given  that  the  student  reported  on  the  pre-survey  that  he  had  no  prior  experience  in  systems 
engineering. 

The  scores  for  the  entire  matched  set  of  students  increased  from  pre-  to  post-,  particularly  for  the  more 
technical  Categories  B  and  C: 


Pre 

Post 

Category  A 

2.76  (1.04) 

3.09  (1.05) 

Category  B 

2.69  (1.08) 

3.43  (0.88) 

Category  C 

1.59  (1.00) 

2.19  (1.27) 

All  categories 

7.04  (2.60) 

8.70(2.26) 

Figure  6:  BFV  Rubric  Means  and  Standard  Deviations-All  Students  (n=54) 

The  increases  were  statistically  significant  at  p  <.  01  for  all  categories  combined,  as  well  as  for  Category 
B  and  Category  C,  but  not  for  Category  A: 

For  combined  scores,  t(53)  =  4.39,  p<.01,  p=.000. 

For  Category  A,  the  category  related  to  understanding  systems  engineering  as  a  problem  solving 
approach,  t(53)  =  1.72,  p>.05,  p=.092. 

For  Category  B,  the  category  relating  to  systems  engineering  as  a  process  involving  complex  systems, 
t(53)  =  4.22,  p<.01,  p=.000. 

For  Category  C,  the  category  related  to  their  understanding  of  systems  engineering  as  a  discipline 
requiring  specific  professional  traits,  t(53)  =  3.25,  p<.01,  p  =  .002. 
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As  the  means  below  show,  students  with  systems  engineering  experience  were  initially  1.32  points 
above  students  without  this  experience  but  by  the  end  the  difference  was  only  .36  between  the  two 
groups. 


Pre-  Course  Case  Study 

7.32  (2.47) 

Post-Course  Case  Study 

8.86  (2.27) 

Table  17:  BFV  Means  and  Standard  Deviations  -  Students  with  SE  Experience  (n=28) 


Pre-  Course  Case  Study 

6.00(2.28) 

Post-Course  Case  Study 

8.50(2.28) 

Table  18:  BFV  Means  and  Standard  Deviations  -  Students  without  SE  Experience  (n=16) 

The  increases  were  statistically  significant  at  the  p  >  .01  level  for  students  both  groups:7  students 
without  prior  SE  experience,  t(15)  =  5.37,  p<.01,  p=.000;  students  with  prior  SE  experience,  t(27)=2.61, 
p<.01,  p=.007. 

Use  of  the  Bradley  Fighting  Vehicle  Case  Study  Analysis  in  the  Course 

Faculty  used  the  BFV  scenario  in  different  ways.  Several  used  it  as  an  instructional  activity— for  instance, 
for  initiating  discussion  on  systems  engineering  and  ethics.  At  the  Air  Force  Academy,  for  example,  the 
PI  reported  that  the  scenario  provided  a  useful  framework  to  introduce  students  to  their  semester 
projects  and  to  the  ethical  questions  that  might  arise.  In  his  report,  he  noted  that  the  instructor  had  had 
the  students  watch  the  entire  film  of  The  Pentagon  Wars  (not  just  the  clips),  leading  to  a  "discussion 
[that]  was  by  far  the  best  SE  related  discussion  the  team  had  all  year.  All  members  were  involved,  even 
the  non-SE  ones.  We  will  very  likely  repeat  the  movie  day  in  the  coming  year.  It  was  good  for 
engineering  ethics..." 

At  Auburn,  the  PI  also  used  the  scenario  to  generate  class  discussion  and  found  that  "it  provided  a  good 
means  of  motivating  a  systems  outlook."  On  the  other  hand,  the  assessment  team  at  Penn  State  felt 
that  the  prompt  did  not  give  enough  guidance  as  to  the  depth  of  the  expected  response. 


2.4  STUDENT  USE  OF  BLOGS 

One  of  the  assessments  was  a  weekly  weblog  ("blog")  post.  Blogs  are  defined  as  "dated  [online]  entries 
made  of  text  containing  news,  commentary  or  reflections,  with  links  to  other  artifacts  such  as  websites, 
photos  or  other  media... [and]  functionality  for  outside  commentary  by  peers,  [teachers  or  others]  at  a 
distance"  (Chen  et  al.,  2005).  Blogs  have  been  used  in  higher  education  contexts  as  knowledge  logs  to 
gather  information  about  specific  ideas  or  topics,  records  of  personal  life,  assessment  tools,  forums  for 
interacting  and  communicating  with  others;  and  platforms  for  task  management  (Sim  &  Hew,  2010).  In 


7  The  n  for  this  test  was  41;  10  students  from  the  post-scenario  group  (originally  n=52)  could  not  be  matched  with 
their  original  systems  engineering  surveys,  which  had  asked  for  student  status  and  systems  engineering 
backgrounds. 
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this  project,  the  weekly  blogs  were  centralized  in  Sakai,  the  content  management  system  provided  by 
Stevens  that  has  a  built-in  blogging  module.  Eight  institutions  used  the  blog  to  track  project  progress  and 
describe  the  research,  acquisitions,  development,  and  testing  phases  of  their  designs. 

The  students  were  asked  to  answer  several  prompts  in  each  of  their  posts: 

•  What  did  you  and  your  group  accomplish  this  week? 

■  Which  SE  competencies  best  align  with  what  you  did  this  week? 

■  What  specifically  did  you  do  in  terms  of  each  of  these  competencies? 

While  each  university  originally  proposed  researching  a  specific  DoD  target  area,  there  were  often 
multiple  team  projects  being  developed  at  each  university  so  that  blog  entries  reflected  the  diversity  of 
the  projects.  At  UMD,  for  example,  one  group  of  students  blogged  about  integrating  two  different  types 
of  software  used  to  model  micro  fluidic  particle  steering  while  another  group  blogged  about  building  an 
energy-efficient  wireless  sensor  network  to  track  intruders  and  monitor  a  secure  area.  At  SMU,  one 
team  researched  developing  a  facial  capture  application  without  facial  markers  while  another  team 
described  the  process  of  developing  custom  gesture  software.  Part  of  the  class  at  the  Coast  Guard 
Academy  worked  on  building  a  flare  tube  redesign  while  the  other  group  engineered  a  hybrid  vehicle 
(mail  truck)  with  an  electric  motor  capable  of  regenerative  braking.  At  other  institutions,  student  teams 
worked  collaboratively  on  one  project. 

In  their  blogs,  the  students  described  the  phases  of  the  systems  engineering  design  process,  including 
researching  initial  product  ideas  and  platforms;  calculating  and  recalculating  their  original  costs; 
purchasing  materials;  meeting  with  customers  to  discuss  product  specifications;  visiting  offsite  locations 
to  learn  about  fabrication  techniques  and  systems  integration;  testing  and  modifying  designs;  and  lastly, 
presenting  before  advisors  and  clients  and  evaluating  the  projects  of  their  peers. 

The  types  of  problems  that  student  teams  described  in  their  blogs  included  making  design  tradeoffs; 
providing  adequate  security  for  their  (wireless)  products;  relying  too  much  on  knowledge  or  technical 
skills  of  one  team  member  with  a  specific  area  of  expertise;  setting  reasonable  and  achievable  goals  for 
product  design  within  a  school  year;  struggling  to  communicate  between  members  of  an 
interdisciplinary  team  with  many  different  perspectives  and  varying  levels  of  expertise;  and  managing 
time  and  design  constraints. 

However,  the  students  took  different  approaches  to  the  weekly  online  postings,  and  answered  them 
with  varied  degrees  of  diligence,  depending  on  the  extent  to  which  their  instructions  enforced  the  use  of 
the  prompts.  At  one  end  of  the  spectrum  were  students  who  wrote  clear,  highly  technical  narrative 
descriptions  of  their  projects  and  tied  these  clearly  to  the  systems  engineering  competencies.  The 
following,  from  Auburn,  is  an  example: 
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Group  3  -  Week  11  (3/31/11  ■»  4/6/11)  ivmon.io^t^^du  20ii  P 

12:<7J7> 

Croup  3  W«k  ll  Update  Blog 

4/4/ n 

Group  43 

Thu  week  wt  btgsn  the  integration  of  our  twq  moo  *ub*y*rm  the  QARSP  i.*abotl  lyKem  and  the  moWe  dene*  Up  until  thr*  perr*  no  eornmunkebon  hoi  octuolly  occurred  between  the  mobile 
device  end  the  QfifiZP  melt  In  addition  to  trese  two  morn  systems  our  ler^tr  •rl-ay.njcturo  was  also  utilised  as  the  ‘mode  men-  in  the  overall  system.  Before  troefrvjon  we  had  previously  sent 
dm-my  data  from  the  motoe  dvrict  to  tn#  server  and  from  the  server  to  the  QIRS?  to  we  were  falny  rompetent  that  Integration  woJd  |0  smootNy  We  began  the  Integrator  process  by  drt  mng 
for  the  final  time  exactly  what  data  we  need  to  transfer  from  one  system  to  the  other  and  how  tfro  data  «txJc  be  formatted  This  allowed  us  to  easily  build  interfaces  Into  both  the  mobile  order 
and  the  QARSP  to  receive  and  process  thn  data  accordingly.  It  was  important  to  loch  at  the  needs  of  both  systems  wtwn  Mtabiltfvng  the  format  for  the  data  so  that  we  axid  ensure  that  needs  of 
both  of  the  systems  were  meet  «Xh  a  minimal  amount  of  post  processing  on  eithe*  end 

Once  a  data  format  had  been  established  the  conwrunlcatton  protocols  were  put  into  place.  The  man  concern  here  was  how  often  pita  woud  be  sent  to  and  polled  from  the  server  W* 
estatfu hed  a  poUtog  rate  that  was  yg mficantiy  higher  than  the  sending  rate  to  ensure  that  new  data  was  rocrrwd  promptly.  With  our  amnumcattor.  protocol  established  we  then  proceeded  to 
teed  actual  data  between  the  systems  At  this  port,  the  data  we  um  was  again  dummy  data  to  ensure  that  we  knew  what  values  iroud  be  received  on  the  opposite  end. 

After  a  successhi  communication  test,  tre  final  step  of  Integration  was  to  send  actual  recucst  from  the  mool*  device  to  the  QAftSP  and  ensure  that  trw  QARSP  operated  correctly.  The  initial  tests 
we  ran  In  dws  manner  were  successful.  At  this  poir*  we  feel  confident  that  the  ■nterface  between  ow  two  systems  is  wertrg  the  way  we  trig inally  planned 

The  goals  for  the  final  weak  of  cycle  three  whl  be  to  perform  a  complete  test  of  the  we  I  re  system  as  well  as  addng  In  a  lew  additional  features.  We  hive  a  fWsa»  mobile  device,  cf  a  afferent  «, 
that  we  want  to  tistall  the  communication  on  as  well  as  updating  the  hard  drive  tn  the  QARSR  system  to  a  solid  state  dnve  to  m  prove  rel ‘ability,  in  addition  cxr  Afl  Drone  system  was  resumed  late 
this  cycle.  Initial  tossing  could  not  determma  If  ail  of  the  issues  had  been  resolved  with  the  Aft  Drone  but  we  will  continue  locking  rta  this  during  the  f  nai  week. 

St  Competences  Covered: 

implemantotpn  progress  made  on  mplwnantaiion  of  the  overall  system,  at  this  pclr*  system  is  to  a  point  where  eaismh*  testing  can  begin. 

Rebate,  tty.  AwaQatriHty,  and  Martsrwteifty  Came  n to  effect  when  deodng  the  fmai  mobile  device  to  •mpsement  the  fyttem  on  at  wall  as  decking  tf  it  was  worth  it  to  update  the  hard  drive 
m  the  system  to  a  solid  state  drive. 

RegurcmenU  Analysts  We  had  to  ensure  that  the  test  results  we  ware  collecting  matched  Che  original  requirements  we  establlteed  at  the  begimng  of  the  project, 
integration  Integration  was  completed  th"*  week  between  the  two  mam  syKems  of  our  project 

interface  management  an  interface  had  to  be  established  and  documented  for  commumcethr  between  the  different  components  of  the  overall  system. 

Software  Cngroeerlng  good  engweemig  practices  had  to  be  applied  to  the  code  we  wrote  to  add  functionality  to  the  system  as  well  os  the  ^glue1'  code  toed  to  connect  the  different 
components  of  the  system. 

Cornmunitatoon/Prabiem  Sol  vng/ Strategic  Theikmg/Rrofesilanal  Ethics  As  always  these  competences  were  snportant  dumg  the  Implementation  and  execution  of  our  project 


Figure  7:  Blog  Example  #1 

Another  example,  while  less  detailed  and  not  written  in  narrative  form,  provided  information  about  the 
current  phase  of  the  project— building  a  hybrid  vehicle— and  also  tied  it  to  the  competencies: 


;RT  T9ID*T19015 

Week 

V 

LDjomct 

[Number 

Question 

1 

Wist  dd  you  end  your  group  Accomplish  to*  week 7 

.Answer 

-  Goneroted  «  MiwJon  Stateroom  ky  the  product  o  A  roeteon  otetemont  may  mil  be  required  to  cover  toe  project  toil  will  define  the  product  •  Conceived  a  new  hybrid  system  toei 
a  lows  o  nigh  efficiency  ICE  to  provide  improved  jrbon  and  hiQhway  performance  1  Schema! c  is  attached)  •  Drafted  an  'Opera tonal  Modes’  statement  to  eaplam  toe  system 
ItnoBon  and  why  It *  superior  to  any  current  Hybnd  or  plugin  system 

1  * 

Vwiich  36  coTteetenoes  align  wrto  what  you  CM  tvs  week? 

Ancwtr 

Awareness  of  04»rent  generations  of  electee  vehicle  products  and  energy  (Conffrxied  study  of  hybrid  architecture*  i  Develop  competence  with  eructured  toote  l  AHP  homework  ) 
Be  aware  of  multiple  fjnetans  in  creatng  a  new  product  (Homeworks  Have  deveiooed  awareness  of  playersj 

3 

Wiat  speafleaby  did  you  do  in  terns  of  e*ch  of  terse  ccnpetencros? 

Anawe 

-  Awareness  cf  the  efficency  and  nteroebons  of  tee  cornporents  of  a  hybnd  vehide  system  •  Awaroresa  of  tea  inofhaenaes  of  current  hybnd  vehioe*  and  plug  In  vertices  ■ 
Awarenoasofteempassibilityof  launching  an  eroranew  vofvcia  as  a  c*mer  tor  upgraded  technology  •  Awareness  cf  the  praacallty  of  ccrveringan  existing  06  M  vehicle 
pelkym  to  tedude  tea  new  power  tram  •  Generated  a  Mission  Statement  ky  toe  product  o  A  meson  statement  may  *911  be  requirod  to  cover  toe  protect  teat  will  derma  toe 
product  •  Conceived  a  re w  hybnc  system  teat  a  tows  a  High  efficiency  ICE  to  provide  improved  urban  and  highway  perbnnarce  (Schematic  is  attached)  •  Drafted  an  “Opera  banal 
Modes'  statement  to  explain  toe  system  function  and  why  rt  s  super  or  to  any  outran!  Hybnd  or  pfejg^n  system 

Figure  8:  Blog  Example  #2 

On  the  other  end  of  the  spectrum,  students  provided  details  about  their  projects  sparingly  and 
answered  the  given  prompts  in  a  cursory  fashion,  either  by  listing  numerical  competencies  or  failing  to 
include  any  competencies  or  narrative  detail: 
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Figure  9:  Blog  Example  #3 

Other  students  did  not  answer  the  given  assessment  prompts  and  instead  used  the  blog  site  as  an  online 
portal  to  store  such  digital  media  items  as  PowerPoint  presentations,  conference  posters,  links  to 
project  documentation  in  Google  Docs,  and  photos  and  videos  of  their  designs.  In  one  case,  students 
provided  brief  updates  and  artifacts  of  SE  documentation,  including  a  bill  of  materials,  Gantt  charts,  QFD 
report,  periodic  design  evaluations  and  types  of  analyses. 

Here  is  an  example  of  an  entry  with  a  link  to  a  GoogleDoc  report: 


Figure  10:  Blog  Example  #4 

This  is  an  example  of  an  entry  that  included  a  photograph: 
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The  Hybrid  Vehicle  teem  has  completed  fabrication  of  a  polycar  bo  mte  safety  guard.  This  guard  will  mitigate  safety  issues  with  the  system,  and 
help  prevent  both  electical  shock  hazards  and  entanglement  hazards  while  still  allowing  the  team  to  keep  an  eye  on  the  system. 

Please  note  the  guard  still  has  a  protective  film  on  it  which  will  be  removed  prior  to  testing. 


Figure  11:  Blog  Example  #5 

Of  all  of  the  student  blog  posts,  the  final  prompts  at  the  end  of  the  capstone  experience  generated  the 
greatest  opportunity  for  reflection.  The  prompts  for  the  final  post  were: 

■  What  were  the  most  important  system-level  trade-offs  that  you  had  to  consider  during  this 
project? 

■  If  you  were  to  start  this  project  over  again,  what  would  you  do  differently? 


Many  students  reported  that  if  they  could  have  changed  anything  about  their  projects,  they  would  have 
set  more  realistic  goals  for  themselves  at  the  beginning  of  the  year,  or  would  have  set  "intermediate 
goals"  on  a  week-to-week  basis,  rather  than  just  "cycle  goals."  Here  is  an  example: 

•  "When  coming  up  with  your  own  project,  never  try  to  do  more  than  you  can  accomplish.  I 

remember  at  the  very  beginning  that  we  wanted  to  do  a  lot  with  our  system.  As  time  [went]  on, 
we  realized  we  were  trying  to  do  too  much." 

Students  discussed  having  to  make  trade-offs  throughout  the  development  process,  including  having  to 
purchase  cheaper  or  substitute  components;  piling  less  requirements  onto  their  original  design; 
changing  development  platforms  because  of  performance  issues;  and  modifying  their  designs  to  meet 
DoD  standards.  Some  students  felt  that  rather  than  attempting  to  design  a  fully  operable  system  or 
product,  they  should  have  designed  a  functional  prototype.  Others  reported  that  they  relied  too  much 
during  the  acquisitions  process  on  the  assumption  that  outside  vendors  would  keep  to  an  efficient 
shipping  schedule,  and  discussed  how  they  would  plan  more  in  the  future  to  accommodate  possible 
delays  into  product  development.  One  student  described  wanting  to  begin  the  integration  process 
sooner: 
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•  “The  lessons  we  learned  from  this  were  not  to  rely  on  the  documentation  of  a  product  as  proof 
that  it  will  actually  do  what  it  says  and  to  start  the  integration  process  as  soon  as  possible,  even 
if  it  is  just  in  a  limited  testing  mind  set.  If  we  were  to  begin  again,  we  would  start  attempting  the 
integration  as  soon  as  possible,  even  if  it  was  just  with  dummy  data,  to  ensure  that  our  interface 
worked  the  way  we  intended." 

Another  student  described  how  creating  and  managing  documentation  was  an  important  aspect  of 
systems  engineering  that  had  not  been  encountered  before  in  other  coursework: 

•  "I  was  not  aware  of  the  amount  of  types  of  documentation  that  a  systems  engineering  project 
required.  The  different  competencies  like  requirements  management  and  verification  and 
validation  showed  how  important  organizational  aspects  are  to  a  successful  project.  I  think  they 
resulted  in  a  higher  quality  project." 

A  theme  in  the  final  blogs  was  teamwork,  with  students  discussing  the  advantages  and  disadvantages  of 
working  in  multidisciplinary  teams: 

•  "Because  it  was  multidisciplinary,  I  feel  it  was  more  like  a  job  in  the  real  world  than  other 
projects.  Sometimes  projects  can  be  self-contained,  but  a  lot  of  times  it  requires  a  wealth  of 
knowledge  and  because  people  come  from  different  backgrounds,  it  is  easy  to  play  off  of  an 
individual's  strengths.  The  only  tasks  that  were  difficult  were  the  ones  not  covered  by  anyone  in 
our  team,  i.e.  software  knowledge.  None  of  us  truly  brought  our  specific  disciplines  into  the 
project  but  because  we  were  all  trained  to  think  differently  and  are  different  people,  it  ended  up 
being  a  great  dynamic." 

Toward  the  end  of  the  semester,  as  students  worked  on  performing  final  tests  on  their  prototypes  and 
working  with  their  teams  to  draft  presentation  documentation,  many  posts  reflected  this  plateau  in 
project  trajectory  -  students  at  multiple  institutions  repeatedly  cited  "communication"  as  an  important 
competency: 

W— fc  of  5/2  Wr  tarns  lO3J»»  J0't  10:3>:5«> 

f  txlcwir>f  the  dry  run  pretenutton  -r  (fivto^eO  then?  wes  »  lot  «n  our  fv»l  report  and  proenution  that  the  pmfeswn  wereot  for.  A  tot  of  effort  jpert  on  r*-wnhr*  the 

docvr'entation  to  meet  Che  enter*  outlined  by  the  predewon  dunnf  the  dry  nn  The  presentation  itself  actually  quite  on  ewa*entnf .  the  comment*  fmm  the  prodesiorj  were  hurt  and  to  the 
porn  and  definitely  caufht  our  frpup  off  |uanj. 

While  «erambtin|  to  prapara  the  document*,  our  caam  alto  mtnafad  to  achieve  2  men  (Mutm  to  prate*  du*  rq  our  final.  The  waefc  prate* ad  our  rvwl  to  the  pro#«aort  and  Our  cuttomer , 
and  thay  taemad  plotted  »'h  the  result*.  Chari  at  flew  out  to  Vlrfoe  to  peasant  and  tha  confer  area  aUo  went  welt. 

The  compatanoas  we  usee  m  our  final  woo*;  26  communication  <  present)  rn,  wnctni  documentation),  7  implementation  Ithe  new  fcsturcsi,  10  wUdittfln  (validaunf  tha  final  system),  and  22 
software  eofneennq. 


Figure  12:  Blog  Example  #6 

Student  teams  posted  weekly  standalone  entries,  as  they  were  assigned  to  do,  but  did  not  use  the 
threaded  discussion  feature  of  the  blog  to  respond  to  each  other's  post.  Only  two  of  the  Pis  and  mentors 
used  the  blog  for  occasional  updates  on  student  progress,  syllabus  changes,  and  course  assignments. 
However,  one  PI  noted  in  his  final  report  that  "few  if  any  students  have  been  viewing  the  information" 
that  he  had  posted.  Only  one  mentor  submitted  a  post,  responding  to  a  student  question  about  how  to 
define  a  needs  statement. 
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2.5  BENEFITS  AND  CHALLENGES  OF  SYSTEMS  ENGINEERING  FROM  THE  STUDENTS’ 

PERSPECTIVE 

Although  the  students  did  not  reach  the  level  of  experts  in  terms  of  their  understanding  of  systems 
engineering  terminology,  they  nevertheless  greatly  appreciated  the  contribution  a  systems  engineering 
perspective  had  made  to  their  projects.  Thus  in  response  to  an  open-ended  question  that  asked  if  what 
they  had  learned  about  systems  engineering  had  benefitted  their  capstone  projects,  all  but  a  small 
percentage  said  that  it  had  done  so  at  least  somewhat,  and  most  said  it  had  definitely  done  so.8  This  was 
more  the  case  for  those  with  prior  systems  engineering  experience  than  for  those  without,  but  79 
percent  of  all  groups  had  a  positive  response: 


All 

(n=55) 

With  SE 
experience 
(n=34) 

With  no  SE 
experience 
(n=25) 

Undergraduates 

(n=44) 

Undergraduates  with 
no  SE  experience 
(n=24) 

Yes 

82% 

88% 

76% 

77% 

71% 

Somewhat 

5% 

6% 

4% 

9% 

8% 

No 

13% 

6% 

20% 

14% 

21% 

Table  19:  Student  Appreciation  of  SE  Approach  on  Their  Capstone  Projects 


Those  who  responded  that  systems  engineering  had  benefitted  their  projects  wrote  about  how  it  helped 
them  systematically  work  through  the  entire  design  and  creation  cycle: 

•  "Yes,  many  of  the  concepts  helped  in  the  original  planning  of  the  project.  It  allowed  us  to 
systematically  go  about  the  design  and  implementation  of  our  system.  While  I'm  sure  we 
would've  worked  everything  out  without  it,  systems  engineering  provided  that  general 
framework  for  designing  our  system." 

•  "Yes.  I  think  our  learning  benefited  us  when  we  were  defining  our  requirements,  developing  our 
conceptual  design,  and  by  giving  us  a  procedural  and  logical  way  to  make  decisions." 

•  "Yes  -  the  most  intangible  positive  effect  that  I  found  was  an  almost  continuous  attention  to  the 
system-level  view  and  the  execution  of  the  systems  engineering  process." 

•  "Absolutely,  when  we  started  we  first  had  to  define  the  problem  through  stakeholder  analysis, 
we  then  created  alternatives,  came  up  with  a  scoring  method,  scored  each  system,  compared 
them  for  tradeoffs,  and  gave  a  recommendation.  We  followed  the  Systems  Decision  Making 
Process  in  order  to  solve  the  problem." 

Those  who  felt  that  it  had  not  been  helpful  were  generally  those  who  worked  only  on  narrow  projects  or 
small  subsystems  rather  than  as  part  of  larger  interdisciplinary  teams.  They  came  from  different 
universities,  so  this  was  not  an  opinion  held  by  others  at  the  same  school: 


s  This  analysis  uses  the  matched  set  for  which  we  have  both  pre-  and  post-surveys  in  order  to  correlate  with  prior 
experience.  Note  again  that  students  from  Penn  State  and  UMD  did  not  have  the  opportunity  to  answer  this 
question;  Penn  State  because  the  full  post-survey  had  not  been  designed  by  the  end  of  the  first  semester,  when 
Penn  State  students  completed  their  course,  and  UMD  because  they  used  the  pre-survey  for  the  post-survey.  In 
addition,  not  all  students  in  the  matched  set  answered  this  question. 
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•  “Not  really,  the  courses  focused  more  on  robotics." 

•  “Not  for  my  project.  There  really  was  not  much  that  went  into  it.  My  group  did  the  flare  tube 
modification  where  it  was  just  a  new  mounting  system  and  did  not  require  any  other  areas  of 
engineering." 

•  I  do  not  think  it  benefited  it.  It  was  not  really  applied. 

However,  the  introduction  of  systems  engineering  had  added  a  level  of  complexity  to  their  Capstone 
projects  that  had  its  own  challenges.  In  response  to  an  open-ended  question  that  asked  the  students  to 
describe  the  most  challenging  aspect  of  their  project,  the  most  common  challenges  overall  were 
managing  the  dynamics  of  a  multidisciplinary  group  and  communication  problems.  Group  organization 
and  communication  were  the  most  commonly  listed  issues,  with  those  with  no  prior  systems 
engineering  experience  more  likely  than  the  other  groups  to  list  communication  issues.  In  the  table 
below,  the  highest  percentage  for  each  issue  is  highlighted  in  gray: 


All 

(n=62) 

With  SE 

experience 

(n=34) 

Without  SE 

experience 

(n=29) 

Undergraduates 

(n=48) 

Undergraduates 

without  SE 

experience 

(n=25) 

Group  organization 
and  work 

23% 

21% 

24% 

25% 

24% 

Communication 

issues 

23% 

18% 

28% 

21% 

28% 

Defining  the 
requirements 

16% 

18% 

14% 

10% 

12% 

Constraints  of  time 
and/or  budget 

15% 

18% 

10% 

15% 

12% 

Technical  problems 

13% 

18% 

7% 

15% 

8% 

Defining  the 
problem 

11% 

9% 

14% 

15% 

16% 

Table  20:  Most  Challenging  Aspects  of  SE  Capstones  from  Students’  Perspective 


Some  of  those  whose  greatest  challenges  were  classified  as  issues  of  group  organization  wrote  about 
the  difficulties  of  managing  large  interdisciplinary  groups: 

•  "Organization  in  such  a  large  team.  9  people  may  not  sound  like  a  lot,  but  9  people  with  up  to  6 
simultaneous  jobs  and  conflicting  schedules  was  a  substantial  task." 

•  "The  most  challenging  aspect  was  working  with  a  lot  of  people  from  different  majors  around 
campus.  It  was  difficult  to  work  together  and  figure  out  the  best  solutions." 

•  "The  most  challenging  aspect  of  the  project  was  working  with  so  many  different  engineering 
disciplines.  A  recurring  problem  was  that  some  of  the  disciplines  were  too  focused  on  their 
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particular  system  and  didn't  look  at  the  project  as  a  whole  and  see  the  effects  their  design 
would  have  on  the  whole  base." 

•  "Communication  between  different  majors  and  finding  time  to  meet  at  the  same  time." 

•  "Getting  people  from  different  academic  backgrounds  to  work  together  towards  the  same  goal." 

•  "Being  able  to  successfully  coordinate  across  disciplinary  boundary.  Work  efficiently  in  such  a 
big  team." 

•  "Collaboration  between  several  different  engineering  disciplines." 

Others  wrote  about  how  their  groups  struggled  because  they  lacked  members  from  other  disciplines: 

•  "When  trying  to  design  an  electronic  system,  without  members  who  are  familiar  in  electronics 
design,  made  the  project  much  more  difficult." 

•  "Not  knowing  anything  about  electrical  or  network  components  and  we  didn't  have  anyone  in 
the  group  who  was  knowledgeable  about  this  which  would  have  helped." 

•  "As  a  team  of  mostly  systems  engineers  with  one  mechanical  engineer  and  one  computer 
scientist,  this  project  was  an  extreme  challenge.  We  would  have  liked  an  electrical  engineer  and 
more  mechanical  engineers  as  we  knew  very  little  about  the  mechanics  of  the  system  and  how 
we  could  physically  transmit  data  in  a  water  source.  However,  this  allowed  us  to  learn  about 
each  of  the  elements  we  would  not  have  otherwise." 

These  issues  came  up  across  universities  and  seem  to  have  been  specific  to  the  group  the  student  was 

assigned  to. 

Closely  related  were  difficulties  with  communication,  in  part  because  some  were  at  a  distance: 

•  "Having  an  outreach  student.  While  our  outreach  student  contributed  greatly  to  the  overall 
success  of  our  system,  it  was  sometimes  difficult  to  explain  the  software  aspects  of  the  system 
without  a  hands-on  approach." 

•  "Communication  between  local  and  distance  members  due  to  communication  channels." 

•  I  am  a  distance  student  and  it  was  not  being  able  to  get  hands  on  lab  time  with  actual  system 
components. 

•  "My  travel,  which  placed  me  in  several  different  time  zones  a  week.  This  caused  me  problems  in 
figuring  out  at  what  time  I  needed  to  dial  into  class  or  team  functions." 

Others  wrote  about  the  difficulty  they  had  defining  the  problem: 

•  "The  initial  problem  statement  was  the  most  challenging  because  we  did  not  have  a  defined 
problem  to  begin  with  that  did  not  already  have  a  solution." 

•  "The  most  challenging  aspect  was  determining  what  the  actual  problem  was  that  we  needed  to 
solve,  and  then  developing  a  scoring  process  that  accounted  for  everything  that  was  important 
for  training." 
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•  “The  initial  problem  definition  was  the  most  challenging.  You  have  to  be  solving  the  correct 
problem  in  order  for  your  work  to  have  any  meaning.  Defining  the  correct  problem  will  save  you 
a  lot  of  heartache  later." 

Closely  related  were  issues  with  defining  and  meeting  the  requirements: 

•  "Fitting  the  project  to  the  requirements,  i.e.  making  it  "cool"  while  still  making  it  meet  the 
expectations  presented  to  us  at  the  beginning  of  the  semester." 

•  "Meeting  the  goals  of  the  pilots,  aircrews,  and  the  Coast  Guard  all  in  one.  In  addition,  producing 
something  that  could  meet  all  of  those  criteria." 

•  "Most  challenging  was  trying  to  determine  what  the  DoD  would  want  from  us.  Even  with  a  set  of 
requirements  that  are  sliding  at  times  we  are  trying  to  determine  what  decisions  are  in  the  best 
interest  of  the  DoD  for  the  particular  need  for  which  we  were  hired.  We  had  to  remember  not 
to  offer  things  that  were  not  requested." 

•  "The  most  challenging  part  of  our  project  was  validating  that  our  project  met  our  derived 
requirements." 

In  addition,  the  time  frame  and  budget  created  constraints  that  were  difficult  for  some  to  overcome: 

•  "Time  management,  we  selected  too  many  features  making  our  project  a  bit  too  ambitious,  and 
therefore  we  struggled  to  make  ends  meet  for  deliverables,  causing  delays  and  lower  results 
than  expectations." 

•  "The  most  challenging  aspect  of  our  project  was  meeting  the  deadlines  we  set  for  ourselves.  We 
got  all  of  our  requirements  done  (except  one)  but  we  were  just  barely  behind  for  most  of  the 
project. 

•  "TIME.  We  were  crunched  to  produce  an  artifact  on  time  that  would  meet  requirements." 

•  "The  most  challenging  aspect  of  the  project  was  the  acquisition  of  materials.  Our  team  was 
instructed  to  try  to  obtain  materials  by  soliciting  donations  from  manufacturers.  Our  attempts 
to  do  so,  however,  were  overwhelmingly  unsuccessful,  leading  to  a  large  delay  in  the  project. 

We  were  forced  to  order  materials  ourselves  and  are  awaiting  reimbursement." 


2.6  STUDENT  PERCEPTION  OF  PRODUCT  SUCCESS 

Despite  the  challenges,  82  percent  of  the  students  from  11  institutions  who  responded  to  the  post¬ 
survey  question,  "Did  your  group  produce  a  product  that  you  would  consider  a  success?"  felt  that  it  was: 


Frequency 

Percent 

Yes 

83 

82.2% 

No 

18 

17.8% 
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101 


100% 


Total 


Table  21:  Percent  of  Students  Who  Considered  Their  Projects  a  Success  -  Global  Data 

Although  there  was  a  range  among  universities,  at  no  university  did  less  than  two-thirds  of  the  students 
feel  that  their  projects  were  a  success.  At  two  universities,  Stevens  and  the  Coast  Guard  Academy,  100 
percent  felt  that  this  was  the  case,  followed  by  Auburn  (93  percent),  UVA  (92  percent),  and  SMU  (83 
percent): 


Frequency 

Percent 

A  FA 

No 

1 

25.0% 

Y 

3 

75.0% 

Total 

4 

100.0% 

AFIT 

N 

2 

66.7% 

Y 

1 

33.3% 

Total 

3 

100.0% 

Auburn 

N 

1 

7.1% 

Y 

13 

92.9% 

Total 

14 

100.0% 

CGA 

Y 

17 

100.0% 

MA 

N 

1 

25.0% 

Y 

3 

75.0% 

Total 

4 

100.0% 

MUST 

N 

4 

26.7% 

Y 

11 

73.3% 

Total 

15 

100.0% 

NA 

N 

1 

33.3% 

Y 

2 

66.6% 
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Total 

3 

100% 

PSU 

N 

5 

33.3% 

Y 

10 

66.7% 

Total 

15 

100.0% 

SMU 

N 

1 

16.7% 

Y 

5 

83.3% 

Total 

6 

100.0% 

Stevens 

Y 

7 

100.0% 

UVA 

N 

1 

7.7% 

Y 

12 

92.3% 

Total 

13 

100.0% 

Table  22:  Percent  of  Students  Who  Considered  Their  Projects  a  Success  -  By  Institution 

Completing  the  project  seemed  to  be  a  likely  but  not  necessary  correlate  of  success,  and  it  therefore 
seemed  possible  that  students  could  consider  their  projects  successful  even  if  they  were  not  completed. 
An  open-ended  section  of  this  question  allowed  students  to  explain  their  answers;  all  but  one  did  so. 

There  were  four  main  reasons  students  considered  their  projects  a  success,  with  some  students  listing 
more  than  one  reason.  The  fact  that  the  project  fulfilled  customer  and  system  requirements  was  the 
most  frequently  cited  reason,  closely  followed  by  the  fact  that  they  had  produced  a  functional 
prototype.  As  expected,  there  was  variation  among  institutions  depending  on  the  project,  problem  area 
and  design  of  the  course.  For  example,  students  at  Penn  State,  who  spent  a  one-semester  capstone 
course  working  through  systems  engineering  modules  and  experienced  systems  engineering  instruction 
delivered  via  lectures  and  guest  speakers,  overwhelmingly  cited  exposure  to  SE  concepts  and  processes 
as  their  reason  for  considering  their  product  a  success,  while  students  at  Stevens  reported  finding 
solutions  to  a  real-life  problem  as  their  top  reason.  Most  of  the  students  at  UVA  who  created  a  haptic 
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glove  cited  having  a  functional  prototype  as  the  reason  for  success,  followed  by  discovering  solutions  to 
real-life  problems.  At  the  Coast  Guard  Academy,  students  most  frequently  cited  fulfilled  requirements  as 
a  measure  of  product  success,  followed  by  providing  solutions  to  real-life  problems:  Why  Students  Felt 
their  Projects  Were  a  Success9 


Produced  a  functional 

prototype 

Used  SE 

concepts/processes 

Fulfilled 

requirements 

Solved  real-life 
problems 

AFA 

4 

Auburn 

2 

2 

5 

1 

CGA 

1 

8 

5 

MUST 

4 

2 

5 

PSU 

2 

8 

2 

1 

SMU 

1 

Stevens 

4 

2 

1 

5 

UVA 

10 

3 

2 

5 

USMA 

1 

1 

2 

USNA 

2 

Total 

24 

20 

28 

19 

Table  23:  Why  Students  Felt  their  Projects  Were  a  Success 


Here  are  some  sample  responses: 

•  "The  most  successful  part  of  the  project  was  the  learning  experience.  In  every  other  engineering 
class  I  have  taken,  we  have  explicitly  focused  on  the  development  of  a  single  component  or 
small  system."  (Used  SE  concepts/processes) 

•  "Our  ultimate  objective  was  to  lower  the  amount  of  fuel  and  water  used  within  the  bases.  We 
achieved  that  goal  by  designing  various  systems  which  work  together  to  lower  those  numbers 
significantly."  (Solved  real  life  problems;  fulfilled  requirements) 

•  "While  the  product  is  only  a  proof  of  concept,  we  were  able  to  show  that  the  basic  design 
elements  of  the  product  would  indeed  work  in  the  field.  Many  of  the  design  elements  were 
results  of  the  client's  feedback  and  with  consideration  of  the  Soldier  in  mind."  (Fulfilled 
requirements) 

•  "[Our  product]  was  manufactured  by  a  Coast  Guard  Unit  and  was  received  very  positively  with 

9  Student  responses  to  this  question  were  not  mutually  exclusive;  several  students  cited  one  or  more  reason  for 
achieving  product  success,  including  those  who  also  noted  that  their  product  had  flaws  or  was  only  a  proof  of 
concept. 
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our  interaction  with  many  different  Coast  Guard  assets."  (Fulfilled  customer  requirements; 
solved  real  life  problems) 

•  "Our  group  offers  a  design  that  will  decrease  convoys  to  Forward  Operating  Bases  by  50%." 
(Solved  real  life  problems) 

•  "We  came  together  and  made  sure  that  our  sub-systems  worked  cohesively.  We  didn't  just 
make  separate  sub-systems  in  the  hopes  that  they  would  sync  up  and  work  together.  We 
created  an  entire  functioning  systems  engineering  solution  that  is  a  huge  success."  (Used  SE 
concepts/processes) 

•  "We  all  worked  together  and  designed  a  product  that  we  were  able  to  test,  demonstrate,  and 
present  to  many  different  companies  and  symposiums  effectively."  (Produced  a  functional 
prototype) 

•  "The  most  successful  aspect  of  the  project  was  the  excellent  group  dynamic  and  time 
management  of  the  team."  (Used  SE  concepts/processes) 


The  eight  students  who  felt  that  their  projects  were  not  a  success  were  spread  across  seven  institutions. 
Their  responses  included  the  same  four  main  reasons.  Not  producing  a  prototype  was  not  the  most 
common  reason  cited,  which  was  lack  of  resources: 


No  physical  prototype 
produced 

Technical  problems 
with  prototype 

Did  not  fulfill 
requirements 

Lack  of  resources, 
budgetary  &  time 
constraints 

AFA 

1 

Auburn 

1 

MUST 

1 

4 

SMU 

1 

UVA 

1 

2 

USMA 

1 

USNA 

1 

1 

Totals 

1 

2 

3 

8 

Table  24:  Why  Students  Did  Not  Feel  Their  Projects  Were  a  Success 


Here  are  some  sample  comments: 

•  "[The  project]  scope  was  changed.  I  think  more  time  and  the  right  resources  were  needed  in 
order  for  the  product  to  have  been  successful."  (Lack  of  resources,  budgetary  and  time 
constraints) 

•  "Our  schedule  is  off  from  the  typical  schedule  and  we  got  a  late  start.  We  are  still  working  on 
our  product."  (Time  constraints) 
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•  "We  completed  all  of  our  objectives,  but  one  component  of  our  project,  the  mobile  surveillance 
platforms,  had  two  parts  in  which  one  was  completed  successfully  and  where  the  other  was  not 
due  to  a  manufacturing  defect  in  the  system."  (Technical  problems  with  prototype) 

•  "We  did  not  develop  the  product  that  we  had  in  mind.  This  was  due  primarily  from  difficulty  in 
ordering  of  parts  and  a  team  member  not  doing  his  part."  (Did  not  fulfill  requirements;  Lack  of 
resources,  budgetary  and  time  constraints) 

In  addition,  some  of  the  students  who  did  consider  their  projects  a  success  nevertheless  noted  that 
there  was  still  work  to  do.  They  stated  that  the  prototypes  were  only  proof  of  concept  and  therefore  not 
fully  operational,  needing  further  testing  and  refinement: 

•  "I  believe  that  we  produced  a  successful  prototype.  It  meets  the  requirements  that  we  set  out  to 
meet,  however  the  overall  quality  of  the  system  was  lower  than  we  would  have  liked  due  to  the 
short  development  time  and  the  limitations  of  the  baseline  software.  However  we  know  enough 
to  design  and  implement  a  production  quality  system."  (Time  and  technical  constraints) 

•  "With  more  time  we  would  have  completed  a  great  amount  of  verification  and  validation 
leading  us  to  a  result  in  which  we  can  place  far  more  confidence."  (Time  constraints) 

•  "At  this  point  [the  haptic  glove]  just  needs  more  gestures  implemented"  (Time  and  technical 
constraints) 

Ranking  of  Items 

The  students  were  asked  to  rank  seven  items  in  terms  of  their  perceived  usefulness  in  learning  about 
and  applying  systems  engineering.  The  items  were: 

•  Faculty  course  lectures 

•  Reading  material  provided  by  faculty 

•  Faculty  advisors 

•  Department  of  Defense  mentor(s) 

•  Industry  mentor(s) 

•  Members  of  my  team  in  my  own  engineering  area 

•  Members  of  my  team  from  other  engineering  areas 

The  table  below  has  the  mean  scores  by  institution.  Since  the  highest  ranking  was  1  and  the  lowest  was 
7  (with  a  N/A  choice  included),  lower  mean  scores  indicate  higher  approval  ratings.  Where  the  means 
are  very  low  (as  with  USMA's  rating  of  faculty  advisors)  or  very  high  (as  with  SMU's  rating  of  team 
members  from  other  engineering  areas),  there  was  a  high  level  of  consistency— in  the  case  of  USMA,  for 
example,  every  student  had  faculty  advisor  as  the  first  choice.  In  addition,  for  some  students  an  item 

could  be  Not  Applicable  (N/A),  which  was  not  counted  in  the  means,  while  other  students  at  the  same 

school  could  rank  that  same  item  a  1  or  2,  presumably  because  some  items  were  applicable  to  some  of 
the  participants  but  not  others.  In  addition,  there  was  often  very  little  consistency  within  a  school.  For 
example,  at  one  school  "team  members  from  other  areas"  was  ranked  first  by  two  students,  second  by 
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two  students,  fourth  by  two  students,  fifth  by  two  students,  and  seventh  by  one  student.  In  this  table, 
the  lowest  and  highest  scores  for  each  institution  are  highlighted  in  green  (lowest)  and  green  (highest):10 


AirForce 

(n=6) 

Auburn 

(n=14) 

CGA 

(n=17) 

MUST 

(n=16) 

SMU 

(n=6) 

Stevens 

(n=7) 

UVA 

(n=13) 

USMA 

(n=4) 

AFIT 

(n=3) 

Course  lectures 

5.40 

3.46 

4.59 

3.31 

3.33 

3.33 

3.89 

4.67 

4.33 

Course  reading  material 

5.50 

4.17 

3.64 

4.57 

3.67 

3.00 

4.82 

3.50 

4.67 

Faculty  advisors 

4.20 

3.29 

3.42 

3.71 

3.83 

4.00 

3.09 

1.00 

1.67 

DoD  mentor(s) 

3.00 

4.27 

4.82 

4.58 

3.00 

3.50 

4.83 

5.75 

4.50 

Industry  mentor(s) 

3.00 

4.91 

3.50 

4.77 

3.50 

5.40 

5.09 

4.00 

4.50 

Team  members  own  area 

2.60 

3.08 

2.94 

2.85 

4.00 

4.43 

3.25 

3.75 

3.00 

Team  members  other  areas 

4.50 

3.33 

3.78 

3.73 

6.00 

4.00 

3.23 

4.75 

4.00 

Table  25:  Ranking  of  Items  From  Students  Perspective 


There  are  a  few  patterns  across  the  entire  set  of  schools.  Team  members  from  the  students'  own 
engineering  areas  were  rated  most  valuable  for  four  of  the  seven  institutions  and  were  consistently 
rated  as  more  useful  than  team  members  from  other  engineering  areas.  In  addition,  industry  mentors 
were  rated  as  the  least  valuable  in  four  of  the  seven  schools. 


2.7  PERCEPTIONS  OF  CHALLENGES:  PRINCIPAL  INVESTIGATORS  VS.  STUDENTS 
Student  Perceptions  of  Challenges 

Students  across  institutions  shared  many  of  the  same  challenges,  regardless  of  their  course  status, 
engineering  background,  or  whether  or  not  they  came  from  military  or  civilian  schools.  They  reported 
challenges  in  defining  a  problem,  organizing  and  communicating  in  an  interdisciplinary  team,  and  having 
to  limit  project  scope  because  of  time  and  budgetary  constraints,  or  because  of  unrealistic  expectations 
or  inability  to  meet  customer  or  system  requirements.  Students  also  described  part  acquisition,  lack  of 
technical  expertise,  systems  integration,  and  fulfilling  changing  requirements  as  other  factors  that  they 
negotiated  in  their  attempts  to  design  systems. 

AFA-  parts  acquisition;  defining  customer  needs  &  requirements 
AFIT-  systems  integration;  delegating  tasks  &  time  management 

Auburn-  parts  acquisition;  systems  integration;  fulfilling  requirements;  time  constraints;  limiting  project 
scope;  organizing  a  large  interdisciplinary  team 

CGA-  lack  of  technical  expertise  in  certain  areas;  problem  definition;  organizing  and  communicating  with 
a  large  interdisciplinary  team;  testing  the  final  prototype 


10  This  is  an  analysis  of  all  responses,  not  the  matched  set.  There  were  no  responses  from  Penn  State  because  this  question  was 
not  added  until  the  final  survey  in  May  and  none  from  UMD  because  they  responded  to  the  baseline  survey  both  pre-  and  post¬ 
implementation. 
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MUST  -  limiting  project  scope;  constantly  changing  requirements;  time  constraints;  lack  of  technical 
expertise  in  certain  areas 

"Most  challenging  was  trying  to  determine  what  the  DoD  would  want  from  us.  Even  with  a  set  of 
requirements  that  are  sliding  at  times  we  are  trying  to  determine  what  decisions  are  in  the  best  interest 
of  the  DoD  for  the  particular  need  for  which  we  were  hired.  We  had  to  remember  not  to  offer  things  that 
were  not  requested." 

"It  was  very  heavy  in  electrical  and  communication  engineering.  I  am  a  mechanical  engineer  so  I  had  to 
immerse  myself  into  several  aspects  of  electrical  operation  and  wireless  communications  in  order  to 
design  my  parts  of  the  project  correctly." 

Penn  -  limiting  project  scope;  parts  acquisition;  fulfilling  requirements;  lack  of  technical  expertise; 
organizing  and  communicating  with  a  large  interdisciplinary  team;  problem  definition;  time  &  budgetary 
constraints;  systems  integration: 

"Although  our  project  is  not  quite  finished,  I  would  consider  it  a  success  because  of  how  much  we 
accomplished  in  such  a  short  period  of  time.  Yes,  one  way  to  define  success  might  be  accomplishing  what 
you  set  out  to  do.  But  to  me,  success  is  also  how  much  you've  accomplished  and  how  close  you  come  to 
reaching  something  desired.  The  most  challenging  thing  was  probably  narrowing  this  large  project  into 
something  much  smaller  that  would  be  achievable  in  a  limited  amount  of  time  with  limited  funds." 

"We  have  designed  a  system  that  does  accomplish  the  main  requirements  that  we  were  given.  Most  of 
the  team  members  are  not  familiar  with  programming  and  electronics.  We  had  to  learn  a  lot  on  the  fly 
and  most  of  the  design  work  was  done  by  very  few  individuals." 

SMU  -  lack  of  technical  expertise  in  certain  areas;  limiting  project  scope;  organizing  a  large 
interdisciplinary  team;  problem  definition 

Stevens  -  organizing  and  communicating  with  a  large  interdisciplinary  team;  allocating  resources  & 
delegating  tasks;  parts  acquisition 

UVA  -  organizing  a  large  interdisciplinary  team;  allocating  resources  &  delegating  tasks;  limiting  project 
scope;  fulfilling  requirements;  lack  of  technical  expertise  in  certain  areas;  systems  integration 

"As  a  team  of  mostly  systems  engineers  with  one  mechanical  engineer  and  one  computer  scientist,  this 
project  was  an  extreme  challenge.  We  would  have  liked  an  electrical  engineer  and  more  mechanical 
engineers  as  we  knew  very  little  about  the  mechanics  of  the  system  and  how  we  could  physically 
transmit  data  in  a  water  source.  However,  this  allowed  us  to  learn  about  each  of  the  elements  we  would 
not  have  otherwise." 

UMD  -  students  did  not  answer  this  question 

Wayne  -  students  did  not  answer  this  question 
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USNA  -  parts  acquisition;  organizing  a  large  team;  allocating  resources  &  delegating  tasks 
USMA  -  problem  definition;  time  constraints 


Principal  Investigator  Perceptions  of  Challenges 

Pis  reported  challenges  related  to  course  scheduling  and  grading.  There  was  little  preparation  time  for 
Pis  to  recruit  many  students  for  the  fall  semester.  Pis  reported  improved  spring  semester  recruitment. 

At  institutions  such  as  CGA,  Stevens,  and  SMU,  Pis  reported  that  there  were  advising/grading 
discrepancies  between  students'  disciplinary  advisors  and  their  Capstone  course  instructors  and  that 
future  RT-19  efforts  should  streamline  course  requirements  and  grading. 

Pis  at  AFA,  Auburn,  Penn,  SMU,  and  USNA  all  discussed  how  Systems  Engineering  was  a  complicated 
topic  with  content  that  was  difficult  to  teach.  Pis  were  challenged  by  having  to  balance  overseeing 
students'  capstone  designs  in  addition  to  instructing  them  in  systems  engineering  concepts.  At  Penn,  for 
instance,  the  PI  reported  that  students  were  able  to  grasp  certain  SE  competencies,  such  as 
requirements  definition  and  communication,  but  were  less  able  to  understand  modeling,  verification, 
and  validation  because  of  lack  of  time  and  the  complexity  and  newness  of  the  material.  Pis  also  cited 
communication  between  mentors  as  another  challenge;  this  ranged  from  no  communication  (if  students 
did  not  have  mentors)  to  infrequent  communication.  One  PI  reported  that  students  misinterpreted 
mentor  feedback  due  to  lags  in  communication.  However,  when  mentors  were  in  regular  contact  with 
students  or  had  the  opportunity  to  visit  them  for  design  reviews,  the  impact  was  reported  as  beneficial. 

Other  challenges  included  team  formation  and  communication  between  on-campus  and  remote 
students;  time  and  workload  management;  and  DoD  problem  areas  that  required  complex  technical 
knowledge. 

Pis  from  each  participating  RT-19  school  reported  the  following  challenges 

AFA-  teams  spent  considerable  time  grasping  the  scope  of  the  project;  too  many  systems  engineering 
managers  on  one  team;  and  students  were  not  good  at  scheduling.  Students  had  initial  problems 
understanding  what  the  requirements  were  and  how  the  Requirement  Traceability  Matrix  connected  to 
their  projects/designs;  students  also  had  to  learn  how  to  work  on  an  interdisciplinary  team  with  roles  as 
Systems  Engineers,  Systems  Engineering  Managers,  and  disciplinary  engineers 

AFIT-  short  time  frame  to  recruit  students;  course  scheduling  issues  -  students  were  committed  to  their 
schedules  in  the  first  semester.  "With  all  students  carrying  four  courses  during  the  fall  quarter,  we  were 
not  able  to  do  much  on  the  design  &  integration  for  this  project.  We  were  able  to  put  purchase  orders 
and  contract  tasks  in  place  to  support  the  effort,  to  include  the  technician  support  for  hardware 
integration  &  flight  testing." 

Auburn  -  recruiting  was  a  challenge  since  students  already  registered;  the  case  study  included 
programming  language  that  was  more  complex  than  students  were  prepared  for;  students  did  not 
experience  DIACAP  (security  standards)  until  late  in  the  semester.  They  could  have  benefited  from 
earlier  introduction  and  identification  of  security  vulnerabilities  in  their  systems.  Content  was  difficult 
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for  students  to  grasp  and  they  came  away  with  basic  concepts  in  SE 

CGA  -  would  like  more  consistency  between  grading  of  capstone  majors;  "we  are  removing  some  of  the 
layers  of  advisors  from  projects" 

MUST  -  "student  recruitment  was  a  challenge  because  of  short  time  frame;"  students  did  not  know 
about  systems  engineering  and  were  therefore  disinterested;  forming  teams  was  a  challenge,  as  was 
communication  between  on-campus  and  distance  students;  students  did  not  have  interdisciplinary 
experience  but  the  issues  were  lessened  by  external  SMEs  so  they  could  focus  on  SE  approach  and  less 
on  technical,  discipline-specific  issues.  The  problem  area  was  difficult  for  students  to  grasp;  professor 
will  try  to  map  lectures  more  closely  to  students'  systems 

Penn  -  students  had  more  deliverables  than  capstone  course;  students  demonstrated  lack  of 
understanding  in  their  presentation  of  verification  &  validation,  but  otherwise  grasped  many  SE 
concepts,  delivered  a  prototype  on  time,  worked  capably  in  teams  and  proved  that  they  had  achieved 
valuable  skills  &  knowledge 

SMU  -  students  had  to  drop  a  requirement  due  to  lack  of  access  promised  by  client;  had  issues  with 
remote  communication  between  clients  &  students;  communication  between  industrial  mentors,  faculty 
&  students  was  not  smooth  and  may  have  been  because  of  mentors'  obligations;  grading  criteria  for 
students  from  different  departments  should  be  more  uniform  next  year.  Also  students  experienced 
changing  requirements  from  clients  and  sometimes  misinterpreted  the  feedback.  Similar  to  Penn: 
students  grasped  some  competencies  very  quickly,  but  more  complicated  ones  like  Verification, 
Validation;  Availability  &  Maintainability;  Modeling  &  Simulation  were  less  well-practiced  because  of 
schedule  constraints. 

Stevens  -  professors  from  different  engineering  departments  experienced  conflict  in  grading  and  course 
expectations  vis-a-vis  the  SE  project,  since  multiple  disciplines  were  involved.  Students  did  not  always 
know  whom  to  respond  to  or  whose  advice  to  follow  (their  SE  professor  or  individual  disciplinary 
advisor) 

UMD-  undergraduate  students  had  difficulty  using  simulation/modeling  software  tools;  Pis  had  trouble 
recruiting  students  in  fall  &  experienced  short  preparation  time 

Wayne-  students  were  too  busy  to  blog  and  felt  like  course  expectations  and  workload  were  at  times 
"overwhelming;"  Pis  had  trouble  with  limitations  imposed  by  school  IRB  protocol  on  administering 
assessments 

USNA  -  Mentors  came  in  too  late  to  help  students;  students  had  trouble  grasping  SE  content;  PI 
delivered  SE  content  to  students  through  DAU  material,  but  it  was  not  applicable  to  their  learning.  They 
had  difficulty  grasping  the  difference  between  requirements  and  specifications.  Students  had  trouble 
working  across  disciplines  in  addition  to  learning  SE  content 

USMA  -  Students  spent  too  much  time  on  stakeholder  analysis  &  requirements  development;  did  not 
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have  as  much  time  to  work  on  testing  physical  prototype;  students  did  not  work  with  industry  mentors 
until  2nd  semester 
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3.0  ANALYSIS  OF  INTEREST  IN  DOD  PROBLEM  AREAS  AND 
SYSTEMS  ENGINEERING  CAREERS 


3.1  STUDENT  UNDERSTANDING  OF  DOD  PROBLEM  AREAS 

One  goal  of  RT-19  was  to  expand  students'  awareness  of  the  number  and  variety  of  Department  of 
Defense  problem  areas  that  require  systems  engineering  expertise.  To  assess  change,  a  question  on 
both  the  pre-  and  post-surveys  asked  the  students  to  list  three  engineering  problems  that  they  believed 
were  currently  being  addressed  by  the  Department  of  Defense.  There  were  93  matched  pre-  and  post¬ 
survey  responses  to  this  question. 

There  were  three  major  changes  from  pre-  to  post-survey.  First,  on  the  pre-survey,  14  percent  of  the 
responses  were  blank,  compared  to  only  6  percent  on  the  post-survey.  Second,  on  the  pre-survey,  25 
percent  of  the  responses  were  too  vague  to  be  considered  responsive  to  the  question— for  example, 
problem  areas  described  as  "training  soldiers,"  "electrical,"  "new  platform  development,"  or 
"awareness."  The  percentage  of  vague  responses  was  reduced  to  16  percent  on  the  post-survey.  Third, 
there  was  a  large  increase  from  pre-  to  post-survey  in  the  percentage  of  students  who  did  not  list  a  DoD 
problem  area  as  such  but  listed  systems  engineering  issues  ("requirements  management," 
"requirements  creep,"  "not  meeting  deadlines").  Only  7  percent  of  the  responses  could  be  categorized 
this  way  on  the  pre-survey  compared  to  20  percent  on  the  post-survey— in  fact,  this  was  the  most 
common  response  by  that  time. 

The  specific  areas  chosen  by  the  students  were  coded  as  follows: 
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Problem  area 

Including... 

Energy-related 

Energy  efficiency,  green  energy,  renewable 
energy,  alternative  energy,  fuel  economy 

Weapons/weapons  systems 

Weapons  acquisition,  missile  systems,  fighter 
planes,  better  guns 

Field  needs 

IED  detection,  troop  protection,  expeditionary 
housing,  water  filtration,  lightweight  armor 

Systems  engineering 

Requirements  management,  requirements 
creep,  system  integration,  not  meeting  deadlines 

Cyber  security/Internet  security 

Cyber  security,  secure  communication,  cyber 
defense 

Communication/communications 

systems 

Communication,  communication  networks,  real¬ 
time  information,  inter-systems  communication 

Autonomous  vehicles 

AUVs,  remote  aerial  vehicles,  military  robotics, 
unmanned  systems 

Humanitarian  assistance 

Humanitarian  assistance,  disaster  relief 

Border  security 

Border  security,  border  defense,  border 
surveillance 

Training  needs 

Troop  training,  virtual  simulations,  immersive 
training 

Other 

RFID,  materials  science,  situational  awareness, 
nanotechnology,  sensors,  etc. 

Table  26:  Student  Perception  of  DoD  Problem  Areas 


The  area  that  increased  the  most  was  the  energy-related  problem  area,  particularly  energy  efficiency 
and  green  energy.  The  area  that  decreased  the  most  was  weapons  and  weapons  systems,  which  were  in 
any  case  mostly  listed  by  the  military  academies  and  decreased  somewhat  as  these  students  shifted  to 
other  problem  areas,  some  more  related  to  their  projects.  Other  areas  only  shifted  slightly  in  one 
direction  or  the  other.  The  following  tables  list  problem  areas  from  highest  to  lowest  percent: 
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Table  27:  Pre/Post  survey  responses  -  Problem  Areas  from  Student  Perspective 


Pre-survey  responses  by 
problem  area 

Post-survey  responses  by 
problem  area 

%  CHANGE 

Number  of 

responses 

Percent  of  all 

responses 

Number  of 

responses 

Percent  of  all 

responses 

Energy  related 

29 

13% 

40 

17% 

4% 

Weapons/weapons 

systems 

28 

12% 

21 

9% 

-3% 

Field  needs 

19 

8% 

7 

3% 

-5% 

Systems  engineering 

15 

7% 

46 

20% 

13% 

Cyber  security/Internet 
security 

15 

7% 

21 

9% 

2% 

Communication 

systems 

12 

5% 

9 

4% 

-1% 

Autonomous  vehicles 

10 

4% 

9 

4% 

0% 

Flumanitarian 

assistance 

9 

4% 

3 

1% 

-3% 

Border  security 

4 

2% 

4 

2% 

0% 

Training  needs 

4 

2% 

3 

1% 

-1% 

Other 

23 

10% 

32 

14% 

4% 

Vague 

57 

25% 

36 

16% 

-9% 

TOTAL 

225 

100% 

231 

100% 

Although  it  might  have  been  expected  that  on  the  post-survey,  at  least  one  of  the  three  items  would 
relate  to  the  problem  area  the  student's  institution  had  chosen  to  work  on,  in  fact  the  students  seem  to 
have  interpreted  the  question  to  mean  "other"  DoD  problem  areas.  For  example,  the  Air  Force 
Academy's  chosen  problem  areas  were  solar  energy  and  low-power  computing,  yet  only  one  student 
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mentioned  solar  power  as  a  DoD  problem  area  and  none  mentioned  low-power  computing.  Similarly, 
MUST  focused  on  immersive  training  technologies,  yet  not  one  student  mentioned  this  as  a  DoD 
problem  area.  Instead,  the  students  tended  to  focus  on  areas  that  are  in  the  news  (cyber  security,  green 
and/or  renewable  energy,  energy  efficiency,  border  security,  humanitarian  assistance,  etc.)  or,  for  those 
in  the  military  academies,  part  of  their  daily  lives  (autonomous  vehicles,  field  safety,  systems  for 
servicing  troops  in  remote  areas,  etc.).  The  one  exception  was  the  Coast  Guard  Academy,  whose 
problem  area  was  green  power  generation  and  whose  students  listed  fewer  weapons  systems  and  more 
energy-related  responses  on  the  post-survey. 


3.2  INTEREST  IN  SYSTEMS  ENGINEERING  CAREERS 


A  major  goal  of  the  RT-19  project  was  to  increase  student  interest  in  systems  engineering.  The  pre-  and 
post-course  surveys  therefore  included  a  question  designed  to  see  if  there  was  a  change  in  this  area.  The 
question  was  as  follows: 

On  a  scale  of  1  to  5,  with  5  being  the  highest,  how  interested  are  you  in  the  following? 

Ql:  Becoming  a  systems  engineer 

Q2:  Becoming  a  systems  engineer  for  the  government 

Q3:  Becoming  a  systems  engineer  for  private  industry 


The  5-point  scale  ranged  from  “Not  at  all"  (1)  to  "Very  much"  (5). 

Since  it  was  possible  that  the  matched  set  of  responses  was  biased  in  one  direction  or  the  other,  we 
compared  the  baseline  (pre-course  survey)  responses  for  the  matched  set  with  the  baseline  responses 
for  the  entire  set.  The  table  below  shows  the  difference  in  terms  of  numbers  of  students.  It  is  important 
to  note  that  26  percent  of  the  respondents  in  the  matched  set  were  from  military  academies  and  74 
percent  were  from  civilian  institutions,  close  to  the  29  percent  and  71  percent  in  the  total  pre-survey 
population: 

Pre-Course  Survey  Responses  on  SE  Career  Interest:  All  Students  and  Matched  Set  of  Responses 


All  students 

Matched  set 

School 

Number 

Percent 

Number 

Percent 

AFA 

8 

2.8% 

6 

6.7% 

AFIT 

3 

1.0% 

3 

3.4% 

Auburn 

38 

13.2% 

5 

5.6% 

CGA 

13 

4.5% 

10 

11.2% 

MA 

4 

1.4% 

4 

4.5% 
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MUST 

18 

6.3% 

12 

13.5% 

PSU 

56 

19.4% 

13 

14.6% 

SMU 

17 

5.9% 

2 

2.2% 

Stevens 

42 

14.6% 

9 

10.1% 

UMD 

23 

8.0% 

12 

13.5% 

UVA 

33 

11.5% 

13 

14.6% 

Total 

288 

100% 

89 

100% 

Table  28:  Pre-Course  Survey  Responses  on  SE  Career  Interest  -  All  Students  and  Matched  Set  of  Responses 

The  pre-survey  means  on  the  career  choice  question  for  the  matched  population  were  higher  for  all 
three  questions  compared  to  the  pre-survey  means  for  the  total  population,  suggesting  that  the 
matched  population  was  somewhat  biased  toward  systems  engineering  in  general: 


Q1 

Q2 

Q3 

General 

Government 

Industry 

All  respondents  (n=288) 

3.42  (SD:  1.44) 

3.03  (SD:  1.45) 

3.11  (SD:  1.44) 

Matched  set  (n=89) 11 

3.71  (SD:  1.34) 

3.22  (SD:  1.36) 

3.45  (SD:  1.31) 

Table  29:  Means  and  Standard  Deviations  for  Systems  Engineering  Career  Choices  by  Careers 


When  split  into  student  groups  by  status  (undergraduate  and  graduate),  the  means  in  the  matched 
population  were  again  higher  than  the  means  for  the  total  group.  In  all  cases,  the  means  for  graduate 
students  were  higher  than  the  means  for  undergraduates.  As  might  be  expected,  undergraduates  were 
more  interested  in  systems  engineering  careers  in  a  general  way  than  in  becoming  a  systems  engineer 
for  a  specific  employer  group.  Graduate  students  and  post-graduates  presumably  already  knew  their 
career  paths: 


Q1 

Q2 

Q3 

General 

Government 

Industry 

All  undergraduates  (n=156) 

3.12  (SD:  1.46) 

2.77  (SD:  1.44) 

2.95  (SD:  1.39) 

Matched  set  undergraduate  (n=73) 

3.66  (SD:  1.38) 

3.18  (SD:  1.37) 

3.36  (SD:  1.34) 

11  Although  there  was  a  total  matched  population  of  93,  cited  elsewhere,  three  students  did  not  answer  this 
particular  question  on  the  baseline  survey,  leaving  a  population  of  89  for  this  question  only. 
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All  graduate  students  (n=118) 

3.75  (SD:  1.32) 

3.31  (SD:  1.41) 

3.25  (SD:  1.47) 

Matched  set  graduate  students  (n=16) 

3.94  (SD:  1.18) 

3.44  (SD:  1.32) 

3.88  (SD:  1.01) 

Postgraduates  (n=10) 

4.00  (SD:  1.41) 

3.50  (SD:  1.58) 

3.50  (SD:  1.58) 

Table  30:  Means  and  Standard  Deviations  for  Systems  Engineering  Career  Choices  by  Class  Status 


Finally,  the  means  for  students  with  prior  systems  engineering  experience  were  again  higher  for  the 
matched  group.  But  whether  for  the  unmatched  or  matched  group,  these  means  were  even  higher  than 
the  means  for  graduate  students  or  postgraduates,  suggesting  that  these  students  in  particular  already 
knew  their  career  paths.  Further  evidence  is  provided  by  the  low  standard  deviations,  which  show  there 
was  little  range  within  the  group: 


Q1 

General 

Q2 

Government 

Q3 

Industry 

All  students:  No  prior  SE  experience  (n=172) 

3.02  (1.46) 

2.74(1.47) 

2.77  (1.43) 

Matched  set:  No  prior  SE  experience  (n=43) 

3.05  (1.38) 

2.81  (1.37) 

2.91  (1.34) 

All  students:  Prior  SE  experience  (n=116) 

4.01  (1.19) 

3.46  (1.33) 

3.62  (1.33) 

All  students:  Prior  SE  experience  (n=40) 

4.45  (0.82) 

3.73  (1.24) 

4.08  (0.97) 

Table  31:  Means  and  Standard  Deviations  for  Systems  Engineering  Career  Choices  by  SE  experience 


The  fact  that  the  matched  population  was  somewhat  more  predisposed  to  systems  engineering  careers 
than  the  total  population  raises  the  possibility  that  there  would  be  less  change  in  this  smaller  population 
than  there  might  be  in  the  population  as  a  whole.  In  addition,  student  interest  was  already  relatively 
high  (at  the  mid-point  or  higher  in  the  Likert  scale),  which  leaves  less  room  for  improvement  than  if 
interest  had  been  low  at  the  start.  It  therefore  seemed  likely  that  the  greatest  improvement  would 
come  within  the  group  with  no  background  in  systems  engineering. 

3.3  POST-SURVEY  RESULTS 

Post-survey  means  for  the  matched  population  as  a  whole  increased  for  Q1  (general)  and  Q3  (industry) 
but  remained  essentially  the  same  for  Q2  (government): 


Q1 

General 

Q2 

Government 

Q3 

Industry 

Baseline 

3.71  (SD:  1.34) 

3.22  (SD:  1.36) 

3.45  (SD:  1.31) 

Post-survey 

3.84  (SD:  1.34) 

3.20  (SD:  1.33) 

3.69  (SD:  1.27) 

Table  32:  Post  Survey  Results-Ql,  Q2,  and  Q3  Means  and  Standard  Deviations.  All  respondents  (n=89) 

For  those  with  prior  systems  engineering  experience,  the  means  for  Q1  (general)  and  Q3  (industry) 
increased,  while  the  mean  for  Q2  (government)  decreased: 
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Ql  Pre-Survey 

4.45  (0.82) 

Q2  Pre-Survey 

3.73  (1.24) 

Q3  Pre-Survey 

4.08  (0.97) 

Ql  Post-Survey 

4.50  (0.78) 

Q2  Post-Survey 

3.55  (1.32) 

Q3  Post-Survey 

4.22  (0.83) 

Table  33:  Post  Survey  Results-Ql,  Q2,  and  Q3  Means  and  Standard  Deviations-SE  Experience  (n=40) 

For  those  without  prior  systems  engineering  experience,  however,  the  means  for  Ql,  Q2,  and  Q3 
increased  with  the  mean  for  Q3  (industry)  increasing  the  most: 


Ql  Pre-Survey 

3.05  (1.38) 

Q2  Pre-Survey 

2.81  (1.37) 

Q3  Pre-Survey 

2.91  (1.34) 

Ql  Post-Survey 

3.26  (1.43) 

Q2  Post-Survey 

2.95  (1.23) 

Q3  Post-Survey 

3.28  (1.33) 

Table  34:  Post  Survey  Results-Ql,  Q2,  and  Q3  Means  and  Standard  Deviations-No  SE  Experience  (n=43) 

Despite  the  increases,  a  paired-samples  t-test  showed  no  significant  differences  between  baseline  and 
post-survey  means  for  either  group  in  all  three  questions.  For  students  with  SE  experience,  the  results 
were  as  follows:  t(39)  =  0.31,  p>.05,  p  =  0.76;  for  interest  in  systems  engineering  careers  in  government 
t(39)  =  -0.88,  p>.05,  p  =  0.39;  or  for  interest  in  systems  engineering  in  industry:  t( 39)  =  0.80,  p>.05,  p  = 
0.43.  For  students  without  SE  experience,  the  results  were  as  follows:  for  general  systems  engineering 
career  interest,  t(42)  =  0.98,  p>.05,  p  =  0.34;  for  interest  in  systems  engineering  careers  in  government, 
t(42)  =  0.57,  p>.05,  p  =  0.57;  and  for  interest  in  systems  engineering  in  private  industry,  f(42)  =  0.98, 
p>. 05,  p  =  0.34. 


However,  means  can  obscure  subtle  changes.  Another  way  to  look  at  the  change  from  pre-  to  post¬ 
survey  is  to  look  at  the  change  in  the  percentage  of  students  at  each  point  in  the  Likert  scale.  For 
example,  for  Ql  (general  systems  engineering  career  interest),  about  65  percent  of  students  on  the  pre¬ 
survey  chose  4  or  5,  indicating  high  interest.  This  percentage  increased  to  about  70  percent  on  the  post¬ 
survey,  with  a  distinct  shift  from  4  to  5.  The  same  was  the  case,  although  to  a  lesser  extent,  for  Q3 
(working  in  industry).  In  the  tables  below,  the  choice  with  the  greatest  change  is  highlighted  in  gray: 


Baseline  % 

Post-survey  % 

1 

12.4 

11.6 

2 

6.7 

4.7 

3 

13.5 

14.0 

4 

32.6 

27.9 

5 

34.8 

41.9 

100.0 

100.0 

Table  35:  Post  Survey  Results-Ql  General 
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Baseline  % 

Post-survey  % 

1 

16.9 

15.1 

2 

11.2 

15.1 

3 

24.7 

23.3 

4 

27.0 

27.9 

5 

20.2 

18.6 

100.0 

100.0 

Table  36:  Post  Survey  Results-Q2  Government 
Q3  (Industry) 


Baseline  % 

Post-survey  % 

1 

15.7 

11.6 

2 

4.5 

4.7 

3 

19.1 

16.3 

4 

40.4 

38.4 

5 

20.2 

29.1 

100.0 

100.0 

Table  37:  Post  Survey  Results-  Q3  Industry 


If  we  break  the  matched  population  of  students  into  three  subgroups— low  (scoring  1-2),  medium 
scoring  (3),  and  high  (scoring  4-5)— we  see  that  there  was  no  consistent  pattern  or  concentration  of 
students  from  one  institution  in  the  low-,  medium-,  or  high-scoring  categories.  For  example,  for  the 
post-survey  scores  for  Question  1  (general  systems  engineering  interest),  the  Coast  Guard  Academy  had 
the  greatest  percentage  in  the  low-scoring  category,  UMD  had  the  greatest  percentage  in  the  medium¬ 
scoring  category,  and  PSU/UVA  had  the  greatest  percentage  in  the  high-scoring  category.  In  fact,  PSU, 
UVA,  and  MUST  had  the  highest  percentage  in  the  high  category  for  all  three  questions.  It  is  notable  that 
the  schools  whose  students  expressed  the  highest  interest  in  systems  engineering  careers  in 
government  were  not  the  military  academies. 
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The  tables  below  look  at  the  percent  that  each  institution  contributes  to  each  category  (low,  medium, 
high).  The  largest  percentage  in  each  column  is  highlighted  in  gray. 


%  low 

%  medium 

%  high 

AFA  (n=6) 

15.8 

0 

4.9 

AFIT  ( n  =3) 

0 

8.3 

3.3 

Auburn  (n=  6) 

10.5 

0 

6.6 

CGA  (n=ll) 

21.1 

25.0 

6.6 

MA  (n=4) 

0 

0 

6.6 

MUST  (n=13) 

10.5 

16.7 

14.8 

PSU  (n=13) 

10.5 

8.3 

16.4 

SMU  (n=2) 

5.3 

8.3 

0 

Stevens  (n=9) 

10.5 

0 

11.5 

UMD  (n=12) 

0 

33.3 

13.1 

UVA  (n=13) 

15.8 

0 

16.4 

Total 

100.0 

100.0 

100.0 

Table  38:  Post  Survey  Results-  Q1  General.  Breakdown  by  Partner  Institution 
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%  low 

%  medium 

%  high 

AFA  (n=6) 

10.7 

0 

7.1 

AFIT  (n=3) 

0 

4.5 

4.8 

Auburn  (n=  6) 

10.7 

4.5 

4.8 

CGA  (n=ll) 

21.4 

9.1 

7.1 

MA  (n=4) 

0 

0 

9.5 

MUST  (n=13) 

10.7 

13.6 

16.7 

PSU  (n=13) 

17.9 

4.5 

16.7 

SMU  (n=2) 

3.6 

4.5 

0 

Stevens  (n=9) 

3.6 

27.3 

4.8 

UMD  (n=12) 

7.1 

22.7 

11.9 

UVA  (n=13) 

14.3 

9.1 

16.7 

Total 

100.0 

100.0 

100.0 

Table  39:  Post  Survey  Results-  Q2  Government.  Breakdown  by  Partner  Institution 
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%  low 

%  medium 

%  high 

AFA  (n=6) 

14.3 

0 

5.6 

AFIT  (n=3) 

0 

5.9 

3.7 

Auburn  (n=  6) 

14.3 

0 

5.6 

CGA  (n=ll) 

19.0 

23.5 

5.6 

MA  (n=4) 

0 

0 

7.4 

MUST  (n=13) 

4.8 

17.6 

16.7 

PSU  (n=13) 

14.3 

5.9 

16.7 

SMU  (n=2) 

4.8 

5.9 

0 

Stevens  (n=9) 

9.5 

11.8 

9.3 

UMD  (n=12) 

4.8 

23.5 

13.0 

UVA  (n=13) 

14.3 

5.9 

16.7 

Total 

100.0 

100.0 

100.0 

Table  40:  Post  Survey  Results-  Q3  Industry.  Breakdown  by  Partner  Institution 


If  we  run  pair-samples  t-tests,  we  find  that  the  increase  from  low-  up  toward  high-scoring  is  statistically 
significant  for  the  low-scoring  group  for  all  three  choices  (Ql,  Q2,  Q3)  and  for  the  high-scoring  group  for 
two  of  the  three  choices  (Q2,  Q3)  but  not  for  any  of  the  medium-scoring  group: 

Low  scoring:  In  the  low-scoring  group,  we  found  a  statistically  significant  increase  in  the  means  from 
baseline  to  post-test  in  all  three  choices,  with  the  significance  for  the  last  two  choices  at  the  more 
stringent  .01  level.  Thus  student  means  increased  significantly  from  baseline  in  general  systems 
engineering  interest  f(14)  =  2.23,  p<.05,  p  =  0.04;  for  interest  in  becoming  a  systems  engineer  for 
government  from  baseline  p<.01,  p  =  0.001;  and  for  interest  in  becoming  a  systems  engineer  for 
industry,  from  baseline:  f(16)  =  3.79,  p<.01,  p  =  0.002). 

Medium  scoring:  For  students  who  were  categorized  in  the  medium-scoring  group,  none  of  the  changes 
were  statistically  significant.  Thus  responses  for  question  1  increased  from  their  pre-  to  post-test  means, 
but  not  significantly:  f(9)  =  1.25,  p  =  0.244).  Student  pre-test  means  for  question  2  decreased  slightly  at 
post-test,  but  not  significantly:  f(17)  =  -0.21,  p  =  0.834.  Finally,  the  group  of  students  in  the  mid-range 
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group  increased  in  their  interest  in  becoming  a  systems  engineer  for  private  industry,  but  again  at  a  level 
that  was  not  statistically  significant:  f(13)  =  1.17,  p  =  0.26. 

High  scoring:  For  students  in  the  high-scoring  group,  there  was  a  mixed  picture,  depending  on  the 
choice.  Thus  baseline  means  for  general  systems  engineering  interest  decreased  in  the  post-test,  but  not 
significantly  f(57)  =  -0.87,  p>.05,  p  =  0.39.  For  systems  engineering  careers  in  government,  baseline 
means  also  decreased  in  the  post-test,  this  time  at  a  statistically  significant  level  f(40)  =  -4.39,  p<0.01,  p 
=  0.00;  as  did  baseline  means  compared  to  post-test  means  in  terms  of  their  interest  in  working  for 
private  industry,  although  this  decrease  was  not  significant  t(5)  =  -1.11,  p>. 05,  p  =  0.27. 

More  research  is  needed  to  better  understand  these  changes  and  the  subtleties  in  movement  in  Likert 
scale  responses. 


3.4  STUDENTS’  REASONS  FOR  ENTERING  THE  SE  PROFESSION 

The  previous  career  question  asked  the  students  if  they  would  choose  a  career  in  systems  engineering, 
but  did  not  say  when.  However,  when  the  students  were  given  an  open-ended  question  that  asked  if 
they  would  choose  a  career  in  systems  engineering  and  to  say  why,  55  (80  percent)  of  the  69  who 
answered  the  question  said  that  they  would,  and  many  indicated  that  this  would  be  sometime  in  the 
future,  after  they  had  spent  time  in  their  own  engineering  track.  Only  14  students  (20  percent)  said  they 
would  not,  generally  because  they  already  had  plans  for  a  career  in  a  specific  engineering  discipline. 

It  was  notable  that  money  was  almost  never  listed  as  a  reason.  The  largest  number  was  drawn  to 
systems  engineering  because  they  liked  being  able  to  see  the  larger  picture: 

•  "Yes,  I  would.  I  would  love  being  a  systems  engineer  working  on  complex  projects  especially  in 
the  DoD  arena.  As  a  SE  I  can  see  the  whole  picture  of  the  project  from  the  beginning  to  the  end 
of  its  life  cycle  in  addition  to  understanding  the  system's  features  inside  out.  Moreover,  as  a  SE,  I 
get  to  work  in  and  with  other  disciplines  and  engineers  that  it  really  helps  me  to  understand  the 
project  from  different  perspectives." 

•  "I  would  choose  a  career  in  system  engineering  because  for  every  project  that  I  am  involved 
with,  I  like  to  have  an  overall  understanding  of  the  system.  A  career  in  engineering  is  not  just 
finishing  my  assignments  and  making  money  for  the  family,  it  is  about  delivering  a  wonderful 
product  that  meets  customer  needs  within  budget  and  meets  the  schedule." 

•  "Yes,  because  I  love  working  on  huge  projects  and  managing  a  whole  lot  of  people.  It's  a  pain 
sometimes,  but  it's  so  rewarding  in  the  end  to  see  the  final  huge  project.  I  learned  a  lot  and  I 
really  am  considering  a  career  in  systems  engineering  because  I  had  a  blast  working  with  all  of 
the  people  in  my  group." 

They  also  liked  the  variety  of  projects  that  face  a  systems  engineer: 

•  "In  the  future  I  may  decide  to  pursue  a  career  in  systems  engineering  because  you're  exposed  to 
a  wide  variety  of  areas,  not  just  one  specific  area.  The  projects  in  systems  engineering  vary 
much  more  than  in  individual  engineering  fields." 
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•  "Yes.  The  projects  are  always  varying,  and  it's  a  more  philosophical  approach  to  engineering. 
Maintaining  the  technical  details  will  be  important,  but  having  the  flexibility  to  think  about 
alternative  designs  based  on  systems  engineering  principles  is  more  enticing." 

And  finally,  they  liked  the  challenges  posed  by  working  on  a  large  system,  the  ability  to  see  the  process 
through  from  beginning  to  end,  and  the  interdisciplinary  and  problem-solving  aspects  of  systems 
engineering: 

•  "Yes,  systems  engineering  is  a  very  diverse  field  with  what  seems  like  a  fast  pace  and  constant 
change.  In  addition,  it  seems  like  a  job  that  would  be  challenging  both  on  an  intellectual  level  as 
well  as  a  practical  level.  All  of  these  reasons  make  systems  engineering  seem  like  a  good  career 
choice  to  me." 

•  "Yes,  systems  engineering  allows  an  individual  to  be  a  part  of  many  stages  in  a  solutions  process. 
This  fact  allows  for  a  robust  and  challenging  experience,  which  appeals  to  me." 

•  "Yes,  as  a  systems  engineer  it's  your  job  to  make  sure  all  of  the  pieces  fit  together  in  a  project, 
which  is  a  challenge  in  itself  and  one  of  the  things  that  separates  good  products  from  bad 
product." 

•  "Yes,  it  is  interesting  to  be  able  to  break  a  project  down  and  analyze  it  based  on  different  factors 
in  order  to  prove  it  is  a  viable  and  cost  effective  project." 

•  "Yes,  I  enjoy  the  detailed  nature  of  the  process  as  well  as  the  satisfaction  of  a  good  complete 
design." 

•  "Due  to  the  fact  that  systems  engineering  is  an  interdisciplinary  field  and  requires  a  thorough 
understanding  of  a  number  of  working  principles,  I  would  choose  a  career  in  it." 

•  "Yes.  I  feel  like  as  an  engineer  I  have  struggled  to  find  my  niche  as  I  like  so  many  different  facets 
of  Mechanical  Engineering  and  engineering  in  general.  SE  allows  me  to  learn  many  different 
fields  and  develop  a  specialization  as  I  develop  as  a  systems  engineer." 

When  the  question  was  posed  in  an  even  broader  fashion,  the  responses  were  even  more  positive.  Thus 
in  response  to  the  question,  "Do  the  approaches  and  models  of  systems  engineering  seem  applicable  or 
useful  to  your  engineering  studies  and  future  plans,"  67  students  answered  and  64  of  them  agreed. 

While  this  might  be  assumed  to  be  the  case  for  graduate  students,  it  was  the  case  for  undergraduates  as 
well.  Only  three  of  the  49  undergraduate  students  who  answered  this  particular  question  did  not 
perceive  systems  engineering  to  apply  to  their  future  career  or  studies.  One  of  the  three  wanted  to  work 
as  a  discipline-specific  engineer,  while  two  did  not  want  to  be  engineers  at  all. 

The  other  46  undergraduates  (94  percent  of  those  who  answered)  cited  one  or  more  reasons  why 
systems  engineering  would  be  applicable  and  useful  to  their  future  career  plans  and  studies,  including: 

•  Practically  applying  systems  engineering  concepts  such  as  requirements  analysis,  lifecycle 
models,  problem  definition,  and  project/risk  management  to  design. 

•  Working  in  interdisciplinary  teams  on  complex,  real-life  problems  with  tangible  customers  and 
outcomes. 
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•  Experiencing  firsthand  the  communication  needs  and  demands  of  their  teams  and  clients. 

The  most  commonly  cited  reason  was  that  systems  engineering  concepts  could  facilitate  the  design, 
research  and  development  process  (they  listed  Requirements  Analysis,  Complex  Systems,  Subsystems, 
Integration,  Life  Cycles,  and  Project  Management),  followed  by  working  on  interdisciplinary  teams: 12 


Use  of  systems  engineering  concepts 

23 

Working  on  interdisciplinary  teams  and  problems 

16 

Gaining  real  life  experience 

14 

Improving  verbal  communication/written  documentation 

5 

Table  41:  Why  Systems  Engineering  Would  Be  Useful  and  Applicable  to  Future  Career  Plans  and  Studies 
Sample  responses  include: 

•  "Yes,  regardless  what  I  do  in  the  future,  systems  engineering  has  given  me  a  broad  perspective 
on  the  application  of  various  fields.  ""Yes  because  projects  are  becoming  more  and  more 
complex  and  today's  engineering  feats  combine  so  many  different  disciplines.  Therefore  it  is  so 
important  to  understand  systems  engineering  and  be  able  to  operate  with  different  disciplines 
to  form  one  cohesive  project." 

•  "Yes,  the  methods  of  tracking  requirements  and  specifications  will  be  extremely  applicable  in 
the  engineering  world  as  design  teams  become  more  global.  The  need  to  share  information  and 
document  clearly  when  and  where  a  decision  oriented  and  why  is  extremely  important  in  a 
multinational  design  system." 

•  "Yes,  they  teach  very  well  how  teams  of  people  from  different  backgrounds  should 
communicate  and  work  together.  In  the  real  job  world  almost  all  teams  consist  of  people  from 
different  academic  backgrounds  so  it  is  very  useful." 

•  "Systems  engineering  will  be  applicable  to  my  future  plans.  I  will  be  joining  the  military  and 
knowing  how  the  various  systems  I  come  into  contact  with  work  together  is  important." 

•  "Future  work  will  require  design  and  integration  of  new  and  legacy  systems.  In  order  to 
accomplish  these  efforts,  an  understanding  of  interfaces,  project  requirements,  system 
functions,  and  their  interdependencies  will  be  needed  to  field  a  system  given  the  anticipated 
constraints  of  budget  and  schedule.  The  SE  approaches  and  models  help  focus  these  efforts." 

•  "The  approaches  and  models  allow  you  to  systematically  outline  requirements  of  the  customer 
and  then  craft  out  steps  to  be  followed  to  achieve  these  requirements.  Where  changes  are 
made  to  any  of  the  design  requirement,  the  models  would  be  modified  accordingly  to  align  with 
the  customer.  Everything  about  systems  engineering  is  essentially  about  the  customer." 

•  "[Systems  engineering  approaches  and  models]  help  me  to  see  the  big  picture  of  the  projects 
that  I  am  involved  in.  Proper  management  of  the  project  saves  time  and  money  because  it 
clearly  defines  the  end  result,  the  testing  and  verification  process." 


12  Student  responses  were  not  mutually  exclusive;  several  students  cited  more  than  one  reason  for  the  relevance 
of  systems  engineering  to  their  future  career  plans  and  studies. 
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•  "I  have  found  the  greatest  success  when  working  with  people  with  different  backgrounds  and 
using  a  systems  engineering  approach.  This  approach  includes  the  management  of  the  project 
throughout  its  entire  life  cycle." 


4.0  RT-19  PRODUCTS:  DISSEMINATION,  COURSE 
MATERIALS,  AND  ONLINE  REPOSITORY 


Products  of  RT-19  can  be  classified  into  several  categories: 

•  Student  products,  which  include  prototypes  and  posters 

•  Faculty  and  research  team  products,  which  include  a  variety  of  course  materials,  including 
assessments;  publications  and  papers;  and  interim  and  final  reports. 

•  The  development  of  a  learning  community  and  online  project  repository 


4.1  STUDENT  PRODUCTS 

One  requirement  of  all  SE  Capstone  projects  was  the  development  of  an  artifact  or  prototype.  The 
production  of  physical  prototypes  (hardware  and  software)  was  a  key  deliverable  designed  to  help  a 
diverse  population  of  students  learn  and  apply  systems  engineering  learning.  Examples  of  student 
prototypes  follow. 


Figure  13:  Stevens  Institute-Expeditionary  Housing 
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Figure  14:  Penn  State-Humanitarian  Assistance/Disaster  Relief  Kit 


Figure  15:  Military  Academy-  Immersive  Training 
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Figure  16:  SMU  -  Immersive  Training 

Video  examples  include: 
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Auburn  University 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/TeamlVideo.wmv 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team2Video.m4v 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team3Video.mp4 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team4Video.avi 

Missouri  S&T 

http://msmovie.mst.edu/public/misc/immersion  vest.wmv 

Southern  Methodist  University 
http://www.youtube.com/watch?v=xoE9y2hiMSA 

Stevens  Institute  of  Technology 
http://www.youtube.com/watch?v=ThYZEw7YNbg 

University  of  Virginia 

http://www.voutube.com/watch?v=N6dwl5marHo&feature=youtu.be 

US  Military  Academy 
http://vimeo.com/27850108 


In  addition  to  these  products,  26  papers  and  posters  were  published  and  presented  at  engineering 
education  and  systems  engineering  conferences  such  as  the  American  Society  of  Engineering  Education, 
INCOSE,  SIEDS  and  NDIA  presenting  the  diverse  approaches  undertaken  to  SE  Capstone  courses,  the 
learning  gains,  student  projects,  and  other  student  outcomes  recorded,  and  the  challenges  addressed 
within  the  particular  types  of  universities  in  which  the  courses  were  implemented. 

Dissemination  of  the  RT19  pilot  program  was  facilitated  through  different  venues.  Partners  were 
encouraged  to  participate  in  conferences,  symposiums,  panels.  The  following  list  shows  the 
participation  of  students  and  faculty  members  in  such  events 

1.  National  Defense  Industrial  Association,  12th  Annual  Science  &  Engineering  Technology 

Conference.  Charleston,  South  Carolina.  June  21-23,  2011 

Student  Poster  Presentations: 

•  Missouri  (5  posters) 

•  Wayne  State  (3  posters) 
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•  Penn  State  (1  poster) 

•  Naval  Postgraduate  School  (lposter) 

2.  American  Society  of  Engineering  Education.  118™  Annual  Conference  &  Exposition.  Vancouver 

B.C. Canada.  June  26-29, 2011 

•  Fostering  Systems  Engineering  Education  through  Interdisciplinary  Programs  and  Graduate 
Capstone  Projects.  Jacques,  David  (Air  Force  Institute  of  Technology) 

•  Integration  of  Systems  Engineering  T raining  Modules  into  Capstone  Courses  across  College 
of  Engineering  Departments.  Ellis,  Darin  (Wayne  State) 

•  SE  Capstone:  Experimental  Learning  in  Distributed  Classroom  Environment  for  Systems 
Engineering  Capstone  Projects.  Corns,  Steve(Missouri  University) 

•  SE  Capstone:  Introducing  Multidisciplinary  Capstone  Design  to  the  United  States  Coast 
Guard  Academy.  Adrezin,  Ronald  (US  Coast  Guard  Academy) 

•  SE  Capstone:  Implementing  a  Systems  Engineering  Framework  for  Multidisciplinary 
Capstone  Design.  Sheppard,  Keith  (Stevens  Institute) 

•  SE  Capstone:  Introduction  of  Systems  Engineering  into  an  Undergraduate  Multidisciplinary 
Capstone  Course.  Nemes,  James  (Penn  State) 

•  SE  Capstone:  A  Pilot  Study  of  14  Universities  to  Explore  SE  Learning  and  Career  Interest 
through  DoD  Problems.  McGrath,  B.,  Lowes,  S.,  Squires,  A.,  Jurado,  C. 

3.  2011  IEEE  Systems  and  Information  Engineering  Design  Symposium.  Charlottesville,  Virginia. 

April  29,  2011 

•  A  Systems  Engineering  Approach  to  Micro  Expression  Facial  Motion  Capture  with 
Structured  Light.  Bruner  W.,  Chakravarthy,  T.,  Jones,  K.,  Kendrick,  R.,  LaManna  D. 
(Southern  Methodist  University) 

•  Multiple  User  Motion  Capture  and  Systems  Engineering.  Colvin,  C.,  Babcock,  J.,  Forrest,  J., 
Stuart,  C.,  Tonnemacher,  M.,  Wang,  W.  (Southern  Methodist  University) 

•  The  Design  of  a  Portable  and  Deployable  Solar  Energy  System  for  Deployed  Military 
Applications.  Tyner,  J.,  Coates,  M.,  Holloway,  D.,  Goldsmith,  Daniels,  C.,  Vranicar,  T.,  Roling, 
J.,  Jensen,  D.,  Mundy,  A.,  Peterson,  B.  (US  Air  Force  Academy) 

•  Rapid  Adaptive  Needs  Assessment  (RANA)  Water  Quality  Kit.  Barham,  S.,  Kazlauskas,  S., 
Reynolds,  R.,  Tabacca,  J.,  Verrilli,  E.,  Zhang,  K.,  Harrison,  P.,  Mathew,  M.,  Louis,  G.  (U  of 
Virginia) 
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•  Hand  Tracking  and  Visualization  in  a  Virtual  Reality  Simulation.  Cameron,  C.,  DiValentin,  L., 
Manaktala,  R.,  McElhaney,  A.,  Nostrand,  C.,  Quinlan,  O.,  Sharpe,  L.,  Slagle,  A.,  Wood,  C., 
Zheng,  Y.,  and  Gerling,  G.  (U  of  Virginia) 

•  Using  Electroactive  Polymers  to  Simulate  the  Sense  of  Light  Touch  and  Vibration  in  a  Virtual 
Reality  Environment.  Cameron,  C.,  DiValentin,  L.,  Manaktala,  R.,  McElhaney,  A.,  Nostrand, 
C.,  Quinlan,  O.,  Sharpe,  L.,  Slagle,  A.,  Wood,  C.,  Zheng,  Y.,  and  Gerling,  G  (U  of  Virginia) 

4.  CDIO,  7th  International  Conference.  Copenhagen,  Denmark.  June  20-23,  2011 

•  System  Engineering  in  Senior  Design  Capstone  Projects.  Rudd,  K.,  Waters,  J.,  O'Mara,  C., 
Flaherty,  C.,  Janssen,  M.  (US  Naval  Academy) 

5.  International  Council  of  Systems  Engineering,  21st  Annual  Symposium.  Denver,  Colorado.  June 

20-23,  2011 

•  Panel:  Integrating  Systems  Engineering  into  Engineering  Curricula  through  Capstone 
Projects.  Beth  McGrath,  James  Nemes,  David  Olwell,  David  Umphress. 

6.  McGrath,  E.,  Nemes,  J.,  Olwell,  D.,  &  Umphress,  D.  (2011).  Integrating  Systems  Engineering 
into  Engineering  Curricula  through  Capstone  Projects.  INCOSE  Insight,  September  2011  - 
Volume  14  Issue  3.  pp.  23-24 

7.  Green  Expeditionary  Housing  in  DOE  Competition 

The  Stevens  SE  Capstone  project  was  leveraged  for  the  creation  of  another  large-scale, 
interdisciplinary  project  addressing  sustainable  building  and  energy.  In  the  U.S.  Department  of 
Energy  Solar  Decathlon  2011  competition,  Stevens  partnered  with  the  New  School  -  Parsons 
School  of  Design  and  Habitat  for  Humanity  as  one  of  20  international  finalists.  The  systems 
engineering  approach  of  the  SE  Capstone  project  was  evident  in  this  entry  to  the  competition, 
which  won  top  honors  in  the  "Affordability"  competition. 


4.2  COURSE  STRUCTURES,  MATERIALS  AND  INTERNAL  ASSESSMENTS 

A  variety  of  methods,  approaches  and  structures  were  employed  in  the  implementation  of  the  courses. 
Appendix  A  summarizes  the  differences  in  type  of  student  (graduate/  undergraduate/mix),  course 
integration,  and  DoD  problem  area.  Eleven  SE  Capstone  institutions  provided  course  materials  in  the 
form  of  lecture  notes,  PowerPoint  slides,  reference  materials,  and  related  artifacts  to  the  central  project 
repository  (the  Sakai  web  site).  Some  of  these  materials  were  created  expressly  for  the  SE  Capstone 
course,  while  others  were  adapted/modified  from  existing  course  materials.  The  collection  of  materials 
can  be  found  at  each  partner  institution's  Sakai  Work-Site  (resources  tool)  in  the  following  location: 

Folder:  RT19  2010-2011 

Sub-folder:  Course  Material 
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The  other  three  institutions  (NPS,  Naval  Academy,  and  USCGA)  did  not  provide  course  materials,  but 
provided  course  syllabi  in  this  folder. 

A  review  of  the  course  materials  yielded  the  following  main  themes  of  the  various  SE  Capstone  course 
implementations: 

1.  Teaching  methodologies  to  deliver  systems  engineering  principles  and  methods-There  are  4 
different  methodologies  identified:  (1)  conventional  lectures  to  deliver  exclusively  SE  fundamentals, 
(2)  SE  fundamentals  integrated  simultaneously  within  the  implementation  of  projects,  (3)  Intensive 
all-day  SE  workshops,  (4)  SE  content  included  in  pre-existing  courses. 

2.  Standards-The  spectrum  of  stakeholders  includes  not  only  clients  but  also  regulatory  bodies,  groups 
of  interest,  etc.  Many  times  overlooking  the  diverse  character  of  stakeholders  has  been  a  main 
factor  on  failing  to  deliver  products;  hence,  making  students  aware  of  the  importance  to  comply 
with  safety,  environmental  and  other  type  of  standards  is  helpful  for  them  to  understand  all 
constraints  involved  in  real  world  complex  projects. 

3.  Course  Management  Systems  [virtual  learning  environmentj-Course  management  systems  used  as 
global  repository  for  coursework  materials  contributes  to  improve  students'  organizational  skills, 
which  are  indispensable  for  well  qualified  systems  engineers. 

4.  SE  Software  applications-Most  of  systems  engineering  principles  are  abstract;  therefore,  software 
applications  to  visualize  systems  architecture  [functional  and  physical  view]  and  integration  become 
indeed  an  invaluable  productivity  tool.  Also  simulation/decision-making  packages  are  important  for 
controlling  cost-schedule-quality  [trade-offs], 

5.  Archive  of  recorded  lectures-This  feature  is  remarkably  effective  to  enhance  learning  on  any  field  of 
knowledge,  providing  students  with  the  opportunity  to  get  a  better  understanding  of  what  is  being 
taught  in  the  classroom  by  eliminating  the  burden  of  constantly  taking  notes  as  they  can  always 
review  the  video  recorded  lectures  from  the  archives 

Appendix  B  includes  a  chart  delineating  the  course  materials  developed/used  by  each  institution  and  the 
corresponding  student  deliverables  and  internal  assessments. 

4.3  ONLINE  REPOSITORY:  SAKAI 

In  order  to  manage  the  large  number  of  documents  disseminated  and  collected  by  RT-19  stakeholders, 
the  research  team  created  a  password-protected,  online  document  and  media  sharing  repository  using 
the  Sakai  content  management  system.  Sakai  is  a  private  collaborative  website  with  a  broad  range  of 
functionalities  such  as  Messages,  Announcements,  Blogs,  Resources  (file  sharing)  intended  to  facilitate 
electronic  document  storage  and  online  collaboration.  This  tool  was  necessary  to  provide  access  to 
documents,  assessments,  archived  WebEx  recordings  of  project  information,  report  templates,  and 
other  documentation,  to  a  variety  of  constituencies,  including  faculty  from  14  institutions,  sponsor 
contacts,  and  others.  The  tool  was  also  intended  as  a  way  to  facilitate  communication  and  sharing  of 
resources  and  lessons  learned  among  Pis  and  RT-19  faculty,  as  each  institution  was  allocated  its  own 
work  site  and  file  location  to  upload  private  and  shared  resources. 
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Figure  17:  Sakai  -  Private  Collaborative  Website 

In  addition,  students  of  most  institutions  used  the  blogging  tool  as  the  vehicle  for  their  weekly  updates 
on  project  progress.  Later  on,  the  blogs  were  suggested  as  a  vehicle  to  facilitate  the  communication 
between  students  and  DoD/Industry  mentors  which  was  an  additional  application  of  this  tool.  Access  to 
the  Partner  Institution  Work-Sites  was  restricted  to  the  ASDR&E  representatives  and  the  research  team 
in  order  to  ensure  compliance  with  IRBs  and  intellectual  property  protections. 


Shared  work-sites  were  created  to  facilitate  the  dissemination  of  information.  These  included. 


1.  Project  News.  Used  for  status  updates.  The  tools  mostly  used  were:  announcements,  messages, 
and  resources  which  served  as  a  repository  for  all  online  meetings  (WebEx  sessions)  and 
corresponding  materials,  e.g.  PowerPoint  slides. 

2.  Publications  &  Dissemination.  Papers  authored  by  Pis  and  The  Research  Team  presented  at 
different  conferences  and  symposiums  were  collected  and  gathered  in  this  work-site. 
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3.  Assessments.  Used  by  the  Research  Team  to  post  materials  related  to  the  implementation  of 
common  assessments. 

4.  July  2011  Workshop.  Although  initially  set  up  to  facilitate  the  planning  of  the  RT19/RT19A 
workshop  (held  in  Washington  DC),  this  work-site  also  compiles  student  products  such  as 
posters  and  videos  showing  prototypes  demos. 

5.  DoD  Problem  Areas.  In  August  2010,  an  official  RT19  kick  off  meeting  was  held  in  Washington 
DC  where  a  group  of  DoD  experts  offered  briefings  on  the  focus  areas  of  DoD  interest.  All 
presentation  materials  were  collected  on  this  work-site  to  serve  as  guidance  for  Pis  of  all  partner 
institutions. 

6.  IRB.  Where  materials  related  to  SERC-Stevens  IRB  approval  were  uploaded. 

Functionalities  such  as  Announcements,  Messages,  and  Resources  (for  file  sharing)  made  Sakai  a 
valuable  project  management  tool  for  the  implementation  of  RT19.  Written  and  screencast  tutorials  on 
the  use  of  Sakai  for  project  needs  were  developed  and  made  available  to  all  SE  Capstone  Teams.  The 
blogging  tool  was  envisioned  as  the  default  location  for  students'  weekly  posts,  to  prove  ease  of  use  for 
the  research  team  to  monitor  this  qualitative  assessment.  Although  the  goal  was  to  empower  the  many 
users  from  a  variety  of  organizations  to  independently  access  Sakai  to  locate  and  upload  materials,  and 
enter  blog  posts,  a  great  deal  of  technical  support  and  facilitation  was  necessary  from  the  start  of  the 
project  to  the  collection  of  Pi's  final  reports. 

As  with  the  introduction  of  any  new  tool,  a  learning  curve  and  ease  of  use  curve  will  precede  habitual 
use.  Sakai,  or  other  online  repositories  like  it,  have  the  potential  to  serve  as  an  online  collaboration  tool 
for  RT-19/RT-19A  and  future  SE  Capstone  participants  to  share  and  disseminate  lessons  learned. 

Similarly,  the  WebEx  videoconference  meetings  were  held  periodically  (roughly  monthly)  to  disseminate 
project  information,  showcase  exemplars,  and  answer  questions.  Archived  recordings  were  posted  on 
Sakai  to  enable  faculty  to  refer  back  as  needed. 
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Figure  18:  WebEx  -  Videoconferencing  Software  Application 
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5.0  PROMISING  PRACTICES 


During  spring  2011,  ASDR&E  representatives  conducted  site  visits  to  all  but  one  RT-19  partner 
institution13.  They  identified  a  set  of  nine  promising  practices— approaches  which  were  present  in 
universities  where  the  site  team  reported  that  students  demonstrated  "improved  communication  and 
awareness  of  the  systems  engineering  process. ..where  they  used  a  different  [i.e.,  more  sophisticated] 
vocabulary  to  describe  their  work. ..and  where  their  understanding  of  domain  specificity  and  its 
contribution  to  the  system  were  [evident]"— and  presented  them  at  Panel  Session  1  during  the  July  11- 
13,  2011,  workshop  (McGrath,  Ardis,  Lowes,  Lam,  &  Jurado,  2011).  These  promising  practices  therefore 
described  the  conditions  or  course  characteristics  which  may  have  contributed  to  or  were  present  in 
institutions  which  produced  students  who  possessed  the  types  of  professional  SE  knowledge  and  skills 
which  aligned  with  DoD's  explicit  or  implicit  needs,  as  observed  by  the  DoD  visiting  team. 

Bransford,  Brown  &  Cocking  give  credence  to  experts'  ability  to  notice  features  and  identify  "meaningful 
patterns  of  information  that  are  not  noticed  by  novices,"  and  state  that,  "experts  have  acquired 
extensive  knowledge  that  affects  what  they  notice  and  how  they  organize,  represent,  and  interpret 
information  in  their  environment ..."  Further,  they  note  that,  "Experts'  knowledge  ...  reflects  contexts  of 
applicability:  that  is,  the  knowledge  is  "conditionalized"  on  a  set  of  circumstances"  (Commitee  on 
Developments  in  the  Science  of  Learning,  2000,  p.  31)  . 

The  promising  practices  noted  by  the  ASDR&E  visiting  team  of  experts,  with  additional  comments  from 
the  ensuing  conference  discussion,  are  as  follows: 

1.  Two-semester  course  sequence. 

Fall  semester  tools/  techniques/  approaches  SE  theory  course,  followed  by  spring  semester  design 
project  course.  Fall  course  should  present  balance  of  "traditional"  SE  approaches  with  automated 
tools/  models/  simulation  techniques. 

2.  Cross-disciplinary  faculty  and  multi-disciplinary  student  teams. 

These  provide  the  best  experience  for  students.  However,  expectations  of  SE  competencies  (depth 
of  knowledge/skill  development)  should  be  different  for  undergraduates  vs.  graduates.  From  the 
student  perspective,  the  "real  life  experience"  (e.g.,  communication,  working  with  people  from 
different  backgrounds)  is  critical. 

3.  Regular,  direct  involvement  of  mentors  with  student  project  teams. 

These  should  be  significant  face-to-face  meetings  (i.e.,  twice  monthly)  with  "on-call"  consultations 
between  meetings— both  for  to  help  with  the  engineering  process  and  to  help  foster  SE  career 
awareness  and  an  appreciation  for  systems  engineering. 

4.  Established  relationships  with  nearby  DoD  commands  and  facilities. 


13  The  ASDR&E  team  did  not  visit  Penn  State  University,  which  had  completed  its  fall  semester  course  by  that  point. 
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5.  Creative  use  of  mentors  from  defense  industry  contractors  and/or  DoD  representative. 

Some  institutions  had  built-in  relationships,  either  through  past  PI  industry  or  military  experience,  or 
had  connections  to  other  military  institutions.  ROTC  involvement  where  possible 

6.  Structured  design  reviews  with  DoD  and  industry  mentors  serving  as  reviewers. 

7.  Use  of  SE  Ph.D.  candidates/advanced  graduate  students  as  project  advisors. 

8.  Creative  imposition  of  technical,  budget,  and  schedule  constraints  by  faculty  to  model  "real  world". 

In  addition,  physical  prototypes  are  considered  important  for  student  motivation,  in  order  to 
demonstrate  products  for  DoD  sponsors,  and  in  order  to  begin  to  pipeline  projects  to  more 
advanced  testing.  Prototypes  illustrate  the  tradeoffs  made  during  the  design  process.  Both  software 
and  "hardware"  prototypes  are  acceptable,  including  decision-making  software. 

9.  For  civilian  institutions  that  have  on-campus  ROTC  units,  established  relationships  with  ROTC  units 
for  requirements  analysis,  use  case  testing,  and  solution  viability. 

These  promising  practices  were  implemented  to  varying  degrees  in  the  RT-19  project.  A  graphical 
representation  of  the  presence  (or  lack  thereof)  appears  as  Appendix  E.  Here  we  discuss  them  in  greater 
detail: 

Two-semester  sequence:  theory  then  practice: 

All  but  three  of  the  RT-19  participating  institutions  included  the  two-course  sequence,  with  two  (PSU 
and  UMD)  implementing  single  semester  Capstone  courses  and  one  (NPS)  encompassing  multiple 
semesters. 

Those  Pis  who  advocated  for  the  two-semester  sequence  felt  that  they  needed  the  first  semester  to 
introduce  the  techniques  and  practices  of  systems  engineering  to  students  from  other  engineering 
disciplines  and  felt  that  traditional  classroom  presentation  and  homework  were  effective  for  this 
purpose.  In  addition,  although  software  packages  for  simulation  and  modeling,  project  management, 
and  system  architecture  were  valuable  tools  for  developing  an  appropriate  concept  of  operations  that 
can  be  effectively  tested  through  implementation  of  real  projects,  the  students  need  time  to  learn  them 
well  enough  to  use  them  effectively.  The  differences  lay  in  when  the  project  work  was  introduced.  For 
example,  Stevens  followed  a  two-course  sequence  heavily  focused  on  project  work  and  design,  with 
"just  in  time"  learning  about  SE  competencies  interspersed  throughout.  Other  schools  followed  the 
more  traditional  sequence.  Where  project  planning  did  not  begin  early  enough,  students  reported 
struggling  with  the  timely  acquisition  of  the  necessary  resources. 

Cross-disciplinary  faculty  and  multi-disciplinary  student  teams: 

Most  systems  engineering  projects  span  multiple  engineering  disciplines  and  both  faculty  and  students 
reported  that  the  multi-disciplinary  nature  of  the  projects  was  challenging  but  also  yielded  more  realistic 
experiences.  Many  students  wrote  that  they  had  learned  about  the  difficulties  of  managing 
interdisciplinary  teams,  thus  experiencing  some  of  the  same  challenges  practicing  engineers  face  when 
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working  on  real-world,  multi-disciplinary  projects.  At  the  same  time,  students  ranked  team  members 
from  other  areas  almost  as  highly  as  team  members  from  their  own  disciplines  in  terms  of  their 
perceived  usefulness.  In  addition,  those  groups  that  lacked  sufficient  expertise  from  other  disciplines 
wrote  about  how  they  had  suffered  as  a  result. 

Almost  all  of  the  other  schools  had  some  element  of  multi-discipline  participation,  but  the  teams  from 
Stevens  Institute  and  the  University  of  Virginia  had  particularly  large  and  diverse  teams.  It  is  notable  that 
students  at  both  institutions  rated  their  projects  as  highly  successful. 

Regular  involvement  of  mentors: 

Mentors  played  a  key  role  in  many  of  the  student  projects.  They  provide  needed  practical  experience, 
and  they  help  students  appreciate  the  value  of  their  project  contributions.  All  SE  Capstone  partner 
institutions  had  at  least  one  DoD  mentor,  and  about  half  had  additional  mentors.  In  some  cases  mentors 
visited  their  student  teams  multiple  times  during  the  project,  but  regular  contact  at  least  via 
teleconference  was  important.  Mentors  who  served  as  clients  facilitated  the  creation  of  a  needs 
statement  and  helped  teams  derive  stakeholder  requirements,  emulating  a  systems  engineering 
approach  in  real  world  conditions.  Also,  Pis  reported  that  mentor  participation  in  design  reviews  was 
particularly  effective  for  validation  of  the  process.  Teams  without  regular  and  frequent  contact  from 
DoD  and  industry  mentors  were,  in  some  cases,  frustrated  by  their  lack  of  involvement  and  in  one  case, 
misinterpreted  direction. 

Established  relationships  with  nearby  DoD  commands  and  facilities: 

In  some  cases  student  projects  were  able  to  exploit  their  proximity  to  DoD  facilities.  These  sites  provide 
demonstrable  proof  of  authentic  problems  and  needed  solutions  and  offered  opportunities  for 
interaction  between  students  and  potential  clients  and  users  of  their  systems. 

Creative  use  of  mentors  from  defense  prime  contractors: 

Defense  contractors  provide  a  different  point  of  view  from  DoD  mentors,  representing  "the  solution 
viewpoint,  as  opposed  to  the  problem  viewpoint  of  DoD  sponsors."  In  some  cases,  contractors  were  able 
to  "save  student  teams  from  exploring  too  many  blind  alleys  as  they  have  often  explored  similar  design 
spaces  in  their  work."  In  other  cases,  mentors  played  multiple  roles,  as  clients  and  as  subject  matter 
experts,  although  this  was  not  always  a  successful  combination. 

Structured  design  reviews  with  DoD  and  industry  mentors: 

Reviews  by  external  experts  are  useful  in  all  engineering  disciplines.  They  are  particularly  helpful  to 
student  teams,  and  in  RT-19  they  provided  a  level  of  experience  that  faculty  sometimes  lacked.  In  their 
most  ideal  implementation,  these  reviews  were  practiced  iteratively,  with  opportunities  for  students  to 
learn  from  previous  mistakes. 

Some  schools,  such  as  the  Military  Academy,  taught  structured  design  reviews  early  in  their  curriculum, 
with  opportunities  to  practice  them  in  several  courses.  Other  institutions  taught  them  as  part  of  their 
Capstone  experience.  Students  who  had  more  experience  with  the  process  before  their  Capstone 
courses  were  better  able  to  focus  on  technical  details  during  the  reviews. 
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Use  of  SE  graduate  students  as  project  advisors: 

In  some  cases  graduate  students  were  very  effective  advisors  of  undergraduate  student  teams.  Although 
graduate  students  have  less  experience  than  faculty,  they  are  closer  to  the  undergraduates  in  age  and 
culture.  This  proximity  helped  in  detecting  communication  problems  and  provided  another  channel  for 
student  teams  to  raise  concerns. 

Creative  imposition  of  technical,  budget,  and  schedule  constraints  and  use  of  physical  prototypes: 

Successful  systems  engineering  projects  must  satisfy  all  their  constraints.  It  is  easy  for  students  to  lose 
sight  of  this  when  they  are  in  the  heat  of  solving  a  particular  problem.  Several  student  teams  dealt  with 
budget  constraints  in  ordering  materials  for  prototypes.  In  some  cases  they  had  to  work  through 
standard  acquisition  processes,  with  several  layers  of  approvals  that  delayed  the  process.  In  other  cases 
they  found  ingenious  ways  to  circumvent  spending  limits  by  breaking  purchases  into  separate 
transactions  and  bargaining  with  suppliers.  Exposing  students  to  a  broad  range  of  constraints  helps  them 
attain  a  more  holistic  view  of  systems  engineering.  As  far  as  the  need  to  build  physical  prototypes,  this 
was  clearly  important  to  the  students,  who  felt  their  projects  were  successful  even  if  the  prototype  was 
not  complete. 

Relationship  with  ROTC  units: 

ROTC  units  provided  some  of  the  context  and  expertise  advantages  discussed  earlier  for  nearby  DoD 
facilities.  For  instance,  ROTC  units  are  sources  of  potential  stakeholders  (e.g.  reservists)  needed  for 
development  and  testing  of  use  case  scenarios. 

These  promising  practices  have  guided  the  selection  and  characteristics  of  the  next  phase  of  research 
being  pursued  in  RT-19A  (Pilot  for  Scaling  Up  and  Sustaining  Effective  SE  Capstone  Practices)  and  their 
presence  and  degree  will  be  examined  more  closely  during  that  project,  both  in  terms  of  varieties  of 
implementation  and  correlations,  if  any,  to  intended  student  outcomes. 
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6.0  FINDINGS  AND  RECOMMENDATIONS 


RT-19,  Research  on  Building  Education  &  Workforce  Capacity  in  Systems  Engineering,  was 
conceptualized  and  designed  as  a  pilot  study  to  develop  a  better  understanding  of  the  impact  of 
differing  course  designs,  structures,  materials,  instructional  practices,  and  other  inputs,  such  as  the 
involvement  of  DoD  and  industry  mentors,  on  student  learning  and  career  interest  in  systems 
engineering.  After  a  competitive  selection  process,  14  institutions,  both  civilian  and  military,  developed 
and  enhanced  courses  configured  in  a  variety  of  ways,  but  which  all  encompassed  key  characteristics:  an 
integrative,  project-based  capstone  course  in  which  students  worked  in  teams,  interacted  with  clients 
external  to  their  academic  institution,  and  practiced  systems  engineering  in  the  development  and  design 
of  physical  prototypes  to  address  one  of  a  given  set  of  authentic,  motivating  DoD  problems. 

The  overarching  goal  of  the  research  was  to  help  inform  the  sponsor  about  making  future  investments  in 
Systems  Engineering  Capstone  courses  in  a  nationwide  scale  up  effort,  e.g.,  what  methods  and 
approaches  lead  to  greater  student  learning  gains  and  greater  SE  career  interest,  particularly  interest  in 
DoD  problems  and  careers.  In  addition,  there  were  three  research  questions  centered  on  student 
outcomes:  student  learning  of  systems  engineering;  student  interest  in  SE  careers;  and  student 
awareness  and  interest  in  authentic  DoD  problems.  The  mixed  methods  approach  aimed  to  gather  pre- 
and  post-course  data  from  students  at  all  participating  institutions  with  the  goal  of  correlating  higher 
levels  of  student  outcomes  with  the  course  characteristics  that  produced  those  outcomes. 

A  vast  amount  of  data  was  collected,  both  the  data  anticipated  from  students,  as  well  as  data  from  PI 
reports,  sponsor  site  visit  teams,  a  July  culminating  workshop,  papers,  posters  and  presentations  by 
faculty  and  students,  and  "performance  assessment  data,"  in  the  form  of  student  prototypes  and 
accomplishments  in  a  variety  of  student  competitions.  Due  to  several  factors,  including  (a)  small  sets  of 
matched  pre-/post  student  responses  at  several  institutions,  making  statistically  valid  correlations 
difficult;  (b)  the  variety  of  metrics  of  student  success,  including  those  we  set  out  to  assess  (definitional 
learning,  growth  in  depth  of  analysis  of  a  case  study  using  SE  knowledge,  and  career  [and  DoD  career] 
interest),  as  well  as  those  which  emerged  over  the  course  of  the  project,  e.g.,  student  success  in  external 
competitions,  prototypes  with  high  potential  for  transition  to  near-term  military  use;  and  (c)  the 
multiplicity  of  variables  (problem  area  selected,  graduate  vs.  undergraduate  vs.  mixed  student 
populations,  duration  of  course,  participation  of  mentors,  and  others),  it  was  not  possible  to  correlate 
specific  university  (or  course)  characteristics  with  student  success.  However,  even  though  the  data  are 
not  as  complete  as  the  research  team  would  have  liked14 ;  it  is  possible  to  draw  some  general 
conclusions  from  the  results. 


14  Compliance  by  all  institutions  with  the  requirement  to  administer  the  three  mandatory,  "common"  assessments  pre-  and 
post-course  to  all  students  was  hampered  by  several  factors:  (1)  staggered  start  and  end  dates  of  SE  Capstone  courses  among 
the  14  institutions;  (2)  lack  of  post-course  data  for  two  institutions  (NPS  and  Wayne  State);  (3)  entry  and  exit  of  new  students  in 
the  second  semester;  (4)  competing  end-of-course  demands,  including  participation  in  external  competitions,  on  faculty  and 
students. 
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The  analysis  of  student  definitions  of  systems  engineering  showed  that  the  participating  students  were 
able  to  use  general  systems  engineering  terminology  almost  as  well  as  experts  but  that  they  still  had 
some  way  to  go  in  employing  more  technical  systems  engineering  language.  However,  those  with  the 
most  to  learn— undergraduates  and  those  with  no  prior  system  engineering  experience— improved  the 
most,  particularly  in  terms  of  technical  language.  In  addition,  the  analysis  of  the  Bradley  Fighting  Vehicle 
case  study  showed  that  students  increased  in  their  ability  to  identify  problems  that  mapped  to  specific 
systems  engineering  competencies,  particularly  those  related  to  the  technical  elements,  but  that  they 
were  less  likely  to  mention  the  "soft"  competencies  like  communication  and  leadership.  The  blogs, 
where  used  well,  showed  students  working  through  the  phases  of  the  design  process  and  struggling  with 
various  technical  and  communication  issues  along  the  way. 

It  was  abundantly  clear  that  students  enjoyed  the  real-world  nature  of  the  projects— both  in  terms  of 
building  an  artifact  that  might  be  used  and  in  terms  of  the  SE  project  context  (budget  constraints, 
interdisciplinary  teams,  experts  as  mentor)— and  that  they  appreciated  the  contribution  that  the 
systems  engineering  perspective  brought  to  their  work.  Although  these  courses  do  not  appear  to  have 
had  a  major  impact  on  the  students'  immediate  career  plans,  it  must  be  noted  that  many  had  their 
immediate  post-college  plans  in  place  and  that  a  large  majority  of  both  undergraduates  and  graduate 
students  believed  that  they  might  choose  careers  in  systems  engineering  sometime  in  the  future. 

Given  the  student  results  described  in  earlier  sections  and  summarized  above,  the  set  of  promising 
practices  as  described  in  Section  5,  and  the  original  goal  of  developing  a  set  of  recommended  models  to 
be  scaled  up  in  engineering  institutions  across  the  U.S.  to  address  the  current  and  projected  shortage  of 
SE  talent  for  DoD  and  industry  workforce  needs,  our  recommendations  for  future  implementations  and 
future  study  include: 

1.  Develop  a  methodology  to  prioritize  and  rank  the  student  attributes  and  outcomes  most  likely  to 
meet  DoD  and  defense  industry  needs  in  the  near  term  (0-5  years)  and  longer  term.  Consider 
attributes  such  as  increased  learning  of  SE;  increased  interest  in  DoD  problems;  increased  interest 
in/commitment  to  DoD  careers;  production  of  a  prototype  with  high  potential  for  military  use; 
student  success  in  SE  competitions;  potential  for  recruitment  to  DoD  of  declared  SE  graduate 
students  vs.  the  undergraduate  engineers  with  less  SE  knowledge/experience,  etc.  Such  a 
ranking/prioritization  will  allow  more  specific  targeting  of  resources  into  those  programs  that 
produce  those  high  priority  outcomes. 

2.  Examine  the  presence,  depth,  and  characteristics  of  implementation  of  the  promising  practices 
through  case  study  analysis  (a  component  of  research  included  in  RT-19A);  correlate,  where 
possible,  to  the  highest  priority  student  attributes  described  in  (1),  above. 

3.  Distill  the  attributes  of  effective  DoD  and  industry  mentor  relationships  through  further  analysis  of 
"what  worked"  and  what  did  not.  Investigate  the  incentives  and  rewards  for  mentors  to  continue 
involvement  with  university  partners.  Qualitative  data  suggest  that  features  of  effective  mentoring 
relationships  include:  clear  boundaries  between  the  roles  of  clients  and  technical  advisors; 
engagement  in  frequent,  iterative,  face-to-face  communication,  and  who  see  benefit  in  the 
development  of  long-term  relationships  with  universities  for  their  own  recruitment,  research,  and 
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other  needs.  Consider  regional  and  national  events  followed,  possibly,  by  online  networking  to 
identify  new  mentors. 

4.  In  coursework  and  in  DoD  mentor  communications,  make  very  explicit  the  goal  of  attracting 
students  to  DoD  careers  in  systems  engineering.  It  cannot  be  assumed  that  students  will  understand 
the  variety  of  careers  and  options  available  to  them  based  on  the  one  project  they  work  on.  The 
DoD  mentor  undoubtedly  plays  a  large  role  in  illustrating  that  the  DoD  has  competent  engineers 
with  important  assignments,  but  a  single  individual  focused  on  a  specific  project  cannot  be  expected 
to  showcase  the  career  opportunities  and  the  wide  range  of  problem  areas  available  for  students 
upon  graduation.  Consider  providing  a  "tip  sheet"  or  online  training  for  mentors  to  prepare  then-l¬ 
and  faculty— to  assist  in  communicating  the  opportunities  available  for  SE  careers  within  DoD. 

5.  Leverage  the  experience  and  expertise  of  the  RT-19  and  RT-19A  to  build  and  expand  a  learning 
community  of  SE  Capstone  stakeholders  (engineering  institutions,  clients,  and  mentors)  through:  a 
public  web  site  and  publication  describing  SE  Capstone  courses,  products,  partners,  and  models  and 
support/encouragement  of  academic  dissemination  at  national  forums  attended  by  potential  new 
scale  up  partner  institutions. 

6.  Consider  piloting  new  approaches  to  sustain  the  SE  Capstone  project,  including  the  creation  of  an 
online  repository  of  potential  DoD  problem  areas  and  clients  along  with  a  "venture  fund"  that  would 
provide  small  grants  of  $5,000-$10,000  for  materials  and  access  to  DoD  problems  and  clients  for 
institutions  that  already  organize  Capstone  projects. 

7.  Publicize  in  relevant  professional  journals,  education  media,  and  the  general  media  the 
contributions  of  SE  Capstone  design  teams  to  the  development  of  solutions  critical  for  our  military 
and  our  nation's  security. 

8.  Conduct  a  longer-term  study  (1-5  years)  tracking  RT-19  participants  and  their  career  choices  and 
employment  trends. 
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7.0  ANALYSIS  OF  FINANCIAL  EXPENDITURES 


The  final  expenditure  report  will  be  issued  by  the  SERC  Director  of  Operations  separately,  upon 
completion  of  processing  of  all  pending  transactions.  However,  as  of  October,  2011,  the  following 
represents  the  major  expenditure  categories  and  expenses  in  the  aggregate  as  of  the  current  date. 


Figure  19:  Financial  Expenditures 
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8.0  CONCLUSIONS,  NEXT  STEPS,  AND  FUTURE  RESEARCH 


RT-19  launched  an  ambitious  and  dynamic  experiment  to  better  understand  academic  models  that  could 
lead  to  the  generation  of  SE  talent  for  DoD  and  industry  needs.  More  than  360  undergraduate  and 
graduate  engineering  and  related  students  were  exposed  to  four  broad,  DoD  problem  areas.  Seventeen 
entries  in  student  competitions  such  as  NDIA,  IEEE/SIEDS  and  the  Solar  Decathlon  resulted  from  RT-19 
student  projects.  Twenty-six  papers  and  conference  presentations  were  published/presented  based  on 
RT-19  research.  SE  Capstone  courses  positively  impacted,  to  varying  degrees,  student  learning  of  SE 
capstone  competencies,  their  interest  in  SE  careers,  and  their  interest  in  DoD  problems. 

As  this  project  was  undertaken  as  a  pilot  to  prepare  for  a  larger  scale  up  effort,  a  proposal  for  a  national 
scale  up  was  developed  in  early  2011.  However,  due  to  budget  uncertainties  related  to  the  Continuing 
Budget  Resolution,  a  small-scale  scale-up  effort  was  launched,  which  includes  11  returning  RT-19 
institutions  and  six  new  partners,  chosen  to  collaborate  with,  and  learn  from,  the  first  cohort.  The  new 
project  (RT-19A)  builds  upon  lessons  learned  from  the  pilot: 

•  All  institutions  conduct  the  SE  Capstone  courses  over  at  least  two  semesters,  with  many  starting 
their  student  projects  earlier  than  they  did  during  RT-19 

•  Strong  mentorship  programs  will  be  used  to  guide  and  motivate  students 

•  Students  will  not  be  required  to  keep  blogs,  but  other  forms  of  social  networking  between 
schools  and  students  will  be  investigated 

•  A  simpler  mechanism  for  assigning  identifiers  to  students  on  pre-/post  student  surveys  will  be 
used  to  ensure  that  more  responses  can  be  correlated. 

An  important  goal  of  the  RT-19A  is  to  discover  best  practices  for  bootstrapping  new  systems  engineering 
capstone  experiences  at  institutions  that  do  not  already  have  them.  In  particular,  partnering 
arrangements  between  institutions  are  being  implemented  and  studied  as  one  approach  to  scaling  up. 

In  addition  to  the  final  report,  two  additional  products  are  planned: 

•  A  public  Systems  Engineering  Capstone  web  site  for  informational  and  document/artifact-sharing 
purposes  to  a  range  of  audiences,  including  DoD  sponsors,  returning  and  new  mentors,  other 
federal  agencies,  universities  seeking  to  replicate  effective  SE  Capstone  courses  and  practices,  and 
others.  A  login  feature  will  provide  password-protected  access  to  confidential  areas  of  the  site  for 
the  RT-19/RT-19A  community. 

•  A  glossy  brochure  including  photos  of  student  teams  and  artifacts  summarizing  highlights  of  RT-19 
findings,  lessons  learned,  exemplars,  and  opportunities  for  collaboration. 
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Additional  avenues  (e.g.  Facebook  Page)  to  foster  cross-collaboration  among  students  are  also  being 
considered.  In  RT-19A,  the  research  team  will  promote  collaboration  among  Pis,  and  the  use  of  tools 
such  as  wikis  will  be  considered  for  increased  collaboration  among  Pis,  mentors,  and  other  stakeholders. 
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Appendix  A:  Course  Structure  and  Foci 


School 

Course/Project 

Description 

DoD  Focus 

#Students 

Auburn  University 

Systems  Engineering 
in  a  Secure  Computing 
Intensive  Environment 

1st  course  [Fall  2010]  is  a  broad-spectrum 
overview  to  systems  engineering.  It 
introduces  major  concepts  using  a  case 
study  of  the  security  architecture  of  two 
open  systems  under  consideration  by  DoD. 

2nd  course  [Spring  2011]  is  an  actual  project 
employing  low-cost,  open-source, 
secure  computing.  The  students 
demonstrate  secure  collaboration  using 

The  Android  open  source  software  stack. 

Improvement  of  computer  systems 
to  enable  secure  data  sharing  among 
complex  systems  at  low  cost. 

Course  material  for  the  1st  course 
delivered  through  presentations 
by  speakers  from  industry  and 
government;  lectures,  and  interactive 
students  activities. 

The  2nd  course  is  a  hands  on  sequel 

In  which  students  1  complete  their 
Defense-focused  capstone  project 

Fall'10:  33 

Spring'll:  17 

Mix  of  CS,  IE, 

And  EE 

On-campus 

And  distance 

education 

Missouri  S&T 
University 

Agile  Systems 
Engineering-Active 

And  Experiential 
Learning  Approach 

1st  Course  [Fall2010] :  Introduction  to 

Systems  Engineering  provides  the  student 
with  basic  understanding  of  main  concepts, 
tools,  and  processes  of  systems  engineering. 

2nd  Course  [Spring2011]:  Physical  Artifact 
Creation  and  Validation.  Development  of 
detailed  design  for  a  wireless  haptic  vest 
with  embedded  sensors.  Students  focused 

on  the  wireless  tech  to  activate  embedded 
sensors  and  mechanical  components 

Immersive  Training  Technologies. 

Subtle  simulation  of  real  battlefield 
scenarios.  Operational  scenarios 
simulate  getting  shot,  getting  hit,  and 
minor  restriction. 

Fall'10:  30 

Spring'll:  30 

Mix  of  ECE, 

ME,  and  AE 

On-campus 
and  distance 

education 

Penn  State 
University 

Interdisciplinary 
Capstone  Design 

Project 

This  is  a  one-semester  course/project 
[Fall2010].  Eight  modules  delivered  by 
systems  engineering  faculty. 

Projects  are  completed  using  the  Bernard 

Expeditionary  Assistance  Kit. 

1.  Water  purification  system 

2.  Power  generation  from 
renewable  energy  sources 

3.  Local  situational  awareness 

Fall'10:  17 

Mix  of  BE,  CE, 

EE,  ME,  IE 
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M.  Gordon  Learning  Factory,  a  lab  providing 
modern  design,  prototyping,  and 
manufacturing  facilities. 

system 

4.  Global  low-bandwidth 

communication  unit 

School 

Course/Project 

Description 

DoD  Focus 

#Students 

Southern 

Methodist 

University 

Leveraging 

Interdisciplinary 

Teaching 

Environments  to 

Research  Immersive 
Training  Environments 

1st  Course  [Fall2010],  students  work  in 
interdisciplinary  teams  to  design  an 
architecture  solution  that  meets  customer 
specifications. 

Winter  break:  10  days  Skunk  Works 
Immersion  Design  Experience.  (IDE) 

2nd  Course  [Spring  2011],  students  continue 
to  work  on  interdisciplinary  teams  to  build 
and  test  a  prototype  of  their  design. 

Immersive  Training. 

The  objective  is  to  improve  existing 
capabilities  in  three  areas:  (1)  fidelity 
of  motion  capture  systems,  (2) 
reduction  of  infrastructure  required 
for  team  based  motion  capture,  and 
(3)  high  resolution  facial  expression 
capture  and  replication 

Fair  10:  75 
Spring'll:  11 
Mix  of  CS,  EE. 
and  ME 

3  PhD  students 
acting  as  TAs 

Stevens  Institute 
of  Technology 

Building  Education 
and  Workforce 

Capacity  in  Systems 
Engineering  through 
Capstone  Design 

Implementation  of  SE  in  capstone  senior 
design  course  [one  year  long] 

A  series  of  Systems  Engineering  all-day 
workshops  delivered  to  introduce  SE 
principles  and  methods  to  all  students. 
Capstone  project  starts  in  Spring  2011. 

Green-Expeditionary  Housing. 

For  a  100  person  FOB  and  3-6  months 
deployment.  Modular  housing  with 
micro-grid  support  for  alternate 
energy  sources,  including  low  impact 
solutions  for  waste  and  water 

Fall'  10:  24 

Spring'll:  19 

Mix  of  EM,  ME, 
EE,  CE,  Civ  Eng, 
A&T 

University  of 
Maryland 

Special  Topics  in 
Systems  Engineering 

This  is  a  one  semester  course  that  is  offered 
twice  over  one  academic  year.  The  goal  of 
this  pilot  is  to  introduce  students  to  SE 
through  hands  on  project  experience. 

[  4  grad  students  providing  assistance  to 
undergrads] 

Focuses  on  low-cost,  low-power 
computers  leveraging  open  source 
technologies.  Supports  integrated 
wireless  sensor  networks,  black  box 
design,  smart  tire  system,  border 
security. 

Fall' 10:  15 

Spring'll:  37 

Mix  of  EE,  CE, 

BE. 

University  of 
Virginia 

Extensible  Systems 
Engineering  Capstone 
Experience 

It  exposes  students  to  the  entire  systems 
engineering  process.  This  will  be 
accomplished  via  two  interdisciplinary 
capstone  projects  over  one  academic  year. 
During  the  2nd  semester  the  two  teams  will 

Project  #1:  involves  a  virtual  reality 
system  for  medical  training. 

Project  #2:  This  project  is  focused  on 
developing  a  mobile,  autonomous, 

Fall'  10:  17 

Spring'll:  16 

Mix  of  SE,  ME, 
CS,  BE,  ECE 
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test  and  evaluate  each  other's  projects. 

water  quality  testing  system. 

2  SE  grad  students  [providing  technical 
support] 

School 

Course/Project 

Description 

DoD  Focus 

#Students 

Wayne  State 
University 

Integrated  Material 
Design  and  Realization 
for  HA/DR  Kits 

This  project  integrates  SE  product 
development  concepts  across  4  courses  at 
the  undergrad  and  graduate  level. 

1  Full  semester  course  (Winter  2011)  plus 
modular  insertion  into  multiple  other 
courses  (start  process  -  Sept  2010) 

Expeditionary  Operations. 

The  projects  focused  on  development 
of  elements  of  HA/DR  kits,  such  as 
solar  oven,  water  purification  system, 
alternative  energy 

Fall'  10:  29 

Spring'll:  16 

Mix  of  ME,  ISE 

AFIT 

[Air  Force 

Institute  of 
Technology] 

Introduction  to 

Systems  Engineering 
Process  and  Design 

This  course  [one  academic  year]  provides  a 
broad  introduction  to  a  systematic  approach 
necessary  for  the  formulation,  analysis, 
design  and  evaluation  of  complex  systems. 
Technical  support  provided  by  the 
Autonomous  Navigation  Technology  Center 
associated  with  the  Department  of  Electrical 
and  Computer  Engineering 

Low-power  computing  for  operations 
in  austere  environments. 

Development  of  a  novel  hybrid 
electric  UAV  for  near  silent,  long 
loiter,  low  energy  operations. 

Fall'  10:  5 

Spring'll:  5 

Mix  of  AE,  SE 

NPS 

[Naval 

Postgraduate 

School] 

Transforming 

Graduate  Education  in 
Systems  Engineering 

A  series  of  8  core  SE  courses  [one  academic 
yearjin  the  masters  curriculum  are  being 
taught  in  a  faculty  team  based  pedagogy, 
with  the  capstone  project  integrated  into 
the  entire  curriculum  as  a  carry  through, 
hands  on  experience.  The  courses  provide  a 
holistic  span  of  education  from  systems 
thinking, ,  quantitative  analysis,  through 
system  design  and  production 

Expeditionary  Operations  and  HA/DR 
Assistance  Kits. 

Development  of  novel,  low  density 
power  supplies,  advanced  materials 
with  low  thermal  and  visibility 
properties,  low  signature 
communication  devices. 

[project  starts  in  January  2011] 

Fall'  10:  38 

Spring'll:  38 

SE 
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This  project  integrates  sequentially  two  SE 
courses  over  one  academic  year.  Students 

Low  Power  Computing 

Fall'  10:  7 

Spring'll:  7 

USAFA 

Capstone  Design 

learn  to  successfully  work  in  a 

A  10  KVA  solar  energy  system  for 

[Air  Force 

Project 

multidisciplinary  team,  to  apply  SE  and 

deployed  operations.  The  end  system 

Mix  of  EE,  CE, 

Academy] 

management  tools,  communicate  project 
details,  and  evaluate  contemporary  military 
issues 

incorporates  smart  grid  technology  to 
facilitate  control  and  integration 
[project  starts  in  Spring'll  semester] 

ME,  SE 

School 

Course/Project 

Description 

DoD  Focus 

#Students 

USNA 

[Naval  Academy] 

Principles  of 

Engineering  Systems 
Design 

The  senior  design  capstone  course  [one 
academic  year]  is  enhanced  with  additional 

SE  sections  based  on  experimental 
coursework.  This  is  an  independent  study 
course  based  on  Defense  Acquisition 
University  courses 

Expeditionary  Ops. 

Portable,  low  power  water 
purification. 

Portable,  renewable  power 
generation,  storage  and  distribution 
[most  of  the  project-centric  work  is 
done  in  the  spring  semester] 

Fall'  10:  16 

Spring'll:  16 

Mix  of  EE,  CE, 

NA,  OE 

USMA  [West 

Point] 

Systems  and 

Engineering 
Management  Design 

This  capstone  course  [two  sequential 
courses  over  one  academic  year] 
emphasizes  SE  in  technology  based 
organizations.  Cadets  examine 
interconnections  between  planning, 
organizing,  leadership,  control,  and  the 
human  element  in  production,  research  and 
service  organizations 

Immersive  Training 

Augmented  Reality:  synthetic  environ, 
decision  analysis  for  optical  &  video 
displays,  high  fidelity  tracking 

Fall'  10:  4 
Spring'll:  4 

Mix  of  SE,  EM, 
and  OR 

USCGA 
[Coast  Guard 
Academy] 

Systems  Engineering 
Capstone 

Enhancement 

This  senior  design  capstone  course  [one 
academic  year]  incorporates  critical 
elements  of  systems  engineering 

Expeditionary  Ops.  Green  Power 
Generation  HA/DR 

Portable  hull  inspection  system. 

Green  electric  power  in  remote  hot 

Fall'10:  20 

Spring'll:  24 
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Cadets  must  defend  their  work  at 

climates. 

Mix  of  Civ  Eng, 

preliminary  and  final  design  reviews 

In  water  remote  propeller  cleaner. 
Hybridization  system  for  fleet 
vehicles. 

EE,  and  ME 

NOTES: 


1.  Sources  include  proposal  documents  submitted  by  universities,  interim  and  final  reports,  and  a  summary  prepared  by  R.  McGahern  for  a  presentation 
at  the  2010  Annual  SERC  Research  Review  conference  Nov.  9-10,  2010. 

2.  The  number  of  students  shown  in  the  table  above  include  only  those  who  are  directly  involved  in  the  whole  capstone  experience  [coursework 
+project], 

3.  Abbreviations : 

EM:  engineering  management,  CE:  computer  engineering,  Civ  Eng:  civil  engineering,  EE:  electrical  engineering,  NA:  naval  architecture,  OE:  ocean 
engineering,  AE:  aerospace  engineering,  A&T:  arts  and  technology,  OR:  operations  research,  IE:  industrial  engineering,  ME:  mechanical  engineering, 
CS:  computer  science,  SE:  systems  engineering,  BE:  biomedical  engineering,  ECE:  electrical  and  computing  engineering,  ISE:  Industrial  &  Systems 
Engineering 


Appendix  B:  Course  Materials/Student  Deliverables/Internal  Assessments 


School 

Course  Materials 

Student  Deliverables 

Internal  Assessments 

Auburn 

University 

SE  Lecture  topics:  conceptual  design,  preliminary  design,  detail 
design,  testing,  open  Source  computing  systems,  acquisition, 
security  certification,  systems  security,  decision  analysis, 
configuration  management,  economics,  real  world  systems 
engineering. 

Standards:  National  Information  Assurance  Training  Standard  for 
Senior  Systems  Managers. 

Course  management  system:  Blackboard™ 

Software  applications:  l-CAIV  [decision  analysis],  Eclipse  &  Papirus 
[SysML  diagram] 

Archive  of  video  recorded  lectures  [for  students  viewing] 

Initial  project  idea,  status  reports, 
preliminary  detailed  design 
presentations,  evaluation  reports, 
final  project 

formative  assessments, 
case  study,  mid-term  and 
final  exams 

SE  Lecture  topics:  system  definition  and  concepts,  requirements 
and  specifications,  dynamic  object-oriented  requirements  system 

FaN'10:  full  set  of  requirements, 
functional  analysis,  cost  estimate, 
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Missouri  S&T 
University 

[DOORS]  presented  by  BOEING  mentor,  functional  analysis  and 
decomposition,  quality  function  deployment  [QFD],  conceptual 
systems  design,  DoD  architecture  framework,  risk 
identification/management,  Sys  Eng  planning,  architecture 
evaluation,  manufacturing  and  disposability,  supportability, 
economic  evaluation,  preliminary  design  review,  reliability,  system 
test  and  evaluation,  trade  off  studies,  modeling  and  simulation, 
detail  design,  optimization  in  design  and  operations,  writing 
specifications. 

Course  management  system:  Blackboard™,  WebEx 

work  breakdown  structure,  risk 
assessment,  technical 
management  plan,  system  physical 
architecture,  specifications. 
Spring'll:  DOORs  database, 
interface  document, 
MOEs/TPMs/Attribute  document, 
scenario  and  integration  testing 
document,  system  validation 

Presentations  performed 
for  mock  reviews. 
Components  of  the  final 
written  project  document 
were  assignments  to 
evaluate  progress 

Penn  State 
University 

SE  Lecture  topics:  systems  engineering  fundamentals,  systems 
requirements  analysis  and  allocation,  systems  architecture, 
problem  solving  in  system  design,  decision  and  risk  analysis, 
introduction  to  project  management,  systems  verification  and 
validation,  introduction  to  systems  thinking 

Reference  material:  NASA  Systems  Engineering  Handbook,  2007 
Course  management  system:  ANGEL  [Penn  State  Management 
System] 

System  requirements  document, 
architecture  design  document, 
conceptual  design  review, 
verification  &  validation  plan,  risk 
mitigation  plan,  preliminary  design 
report,  critical  design  review. 

pre-post  surveys,  case 
study 

School 

Course  Materials 

Student  Deliverables 

Internal  Assessments 

Stevens 

Institute  of 
Technology 

All  day  workshops  to  introduce  students  to  key  SE  principles  and 
methods. 

System  level  architecture,  subsystem  level  architecture,  logistics 
and  life  cycle  support,  subsystem  integration  and  test,  system  level 
integration  test. 

NOTE:  Just  the  1st  workshop  was  conducted  as  planned.  The  rest  of 
topics  were  injected  in  an  ad-hoc  and  informal  way  throughout  the 
semester. 

Course  Management  System:  Google  Groups,  Google  Docs, 

Dropbox 

SE  Software  applications:  Labview,  Solidworks 

ConOps  document  including 
overall  problem  analysis,  key 
requirements,  operational 
scenarios,  concepts  for  key  sub¬ 
systems.  Camp  performance 
simulations.  Budget  spreadsheets. 
Presentations. 

interim  reports, 
presentations,  surveys, 
rubric  to  assess  SE 
competencies 
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University  of 

SE  Lecture  topics:  introduction  to  systems  engineering,  strategies 
of  SE  development,  foundations  for  model-based  systems 
engineering,  modeling  system  structure  and  system  behavior, 
object  and  component  based  development,  multi-objective  trade 
studies,  requirements  engineering,  systems  engineering  with  UML 

Lab  assignments, 
instructors'  observations, 

Maryland 

and  SysML  [Sandy  Friedenthal  from  Lockheed  Martin  delivered  a 

final  project 

special  lecture  on  SysML,  which  was  recorded],  system  level 
design,  basic  approaches  to  system  validation/verification,  basic 
approaches  to  system  validation/verification. 

Course  management  system:  UMD's  Institute  for  Systems 
Research-Website 

SE  software  applications:  ParaMagic™  v!6.6  spl,  Matlab/simulink 

presentations. 
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School 

Course  Materials 

Student  Deliverables 

Internal  Assessments 

Wayne  State 
University 

Lecture  topics:  introduction  to  systems  engineering,  concept 
evaluation  and  selection,  risk  management  in  design 

The  above  SE  principles  and  methods  are  complemented  with  the 
following  courses: 

1.  Integrated  Product  Development  -  to  educate  students 
about  the  importance  of  concurrent  and  collaborative 
engineering  in  a  global  economy. 

2.  Thermal-Fluid  System  Design  -  with  emphasis  on 
alternative  energy  tech 

Course  management  system:  Blackboard™ 

presentations,  final 
reports,  SE  Student 

Query,  Interview,  and 
Response  Tool  [SE- 
SQUIRT] 

AFIT 

[Air  Force 
Institute  of 
Technology] 

Lectures  topics:  intro/process  overview,  conceptual  system  design 
and  requirements  definition,  model  based  SE  ,  utility  theory, 
preliminary  system  design,  detailed  design  and  development, 
system  test,  evaluation  &  validation,  reliability,  maintainability  & 
supportability,  affordability,  usability/human  system  integration. 
These  topics  were  complemented  by  3  case  studies 

Standards:  DoD5000,  JCIDS,  DAG 

Reference  Material:  INCOSE  Handbook 

Course  management  system:  Blackboard™ 

SE  software  applications:  Enterprise  Architect,  LEGO  Mind-storm 
robotics  kits 

concept  definition  [ConOps 
document],  architecture 
development,  and  requirements 
traceability 

homework  assignments, 
exams,  case  study 
discussions,  final 
presentations,  final 
reports 
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School 

Course  Materials 

Student  Deliverables 

Internal  Assessments 

USAFA 
[Air  Force 
Academy] 

Lectures  topics:  Introduction  to  DFEC  Capstone  Design  Course, 
Requirements  development,  planning  and  scheduling,  functional 
analysis  and  allocation,  risk  management. 

Course  management  system:  MS-Sharepoint 

Software  applications:  MS-Project,  Crystal  Ball 

The  course  was  modeled  as  an  Air 
Force  Acquisition  project  with 
students  preparing  and  delivering 
several  presentations  at  various 
acquisition  milestones(PDR,  CDR, 
SVR,  etc) 

Course  specific  rubrics 
are  used  at  each  student 
presentation  (PDR,  CDR, 
SVR,  etc) 

USMA  [West 
Point] 

Lectures  topics:  systems  thinking,  stakeholder  analysis,  functional 
analysis,  value  modeling,  value  modeling  workshop,  in-progress 
review  [IPR],  modeling  and  simulation,  VBS2  Lab,  0*Net  Analysis, 
alternative  generation,  solution  enhancement. 

Course  management  system:  SharePoint 

Software  applications:  VBS2,  VenSim. 

several  in-progress  reviews 
throughout  the  year,  final  briefing 
given  to  the  client,  final  report 

literature  review,  in¬ 
progress  reports  and 
briefings,  peer  evaluation, 
capstone  competition 
judging  rubrics,  and 
technical  reports 

USCGA 
[Coast  Guard 
Academy] 

Lectures  topics:  design  process  overview,  problem  definition  and 
need  identification,  quality  function  deployment,  concept 
generation,  functional  decomposition,  evaluation  [Pugh's  matrix], 
codes  and  standards,  human  factors,  design  for  manufacture, 
design  for  assembly  &  recycling,  engineering  economics,  detail 
design,  engineering  ethics,  modeling  and  simulation,  risk- 
reliability-safety,  quality-robust  design-optimization 

Course  management  system:  Blackboard™ 

Software  Applications:  Solidworks 

assignments,  written  and 
oral  reports,  project 
advisors  evaluation 

Abbreviations: 
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SE:  systems  engineering,  DAU:  defense  acquisition  universities,  INCOSE:  international  council  on  systems  engineering,  PDR:  preliminary  design  review, 
CDR:  critical  design  review,  SVR:  system  verification  review. 
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Appendix  C:  List  of  Web-Links  to  videos  of  prototype  demos 

•  Auburn  University 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/TeamlVideo.wmv 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team2Video.m4v 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team3Video.mp4 

http://swemac.cse.eng.auburn.edu/~umphrda/sysEng/RT19/Team4Video.avi 

•  Missouri  S&T 

http://msmovie.mst.edu/public/misc/immersion  vest.wmv 

•  Southern  Methodist  University 
http://www.youtube.com/watch?v=xoE9y2hiMSA 

•  Stevens  Institute  of  Technology 
http://www.youtube.com/watch?v=ThYZEw7YNbg 

•  University  of  Virginia 

http://www.voutube.com/watch?v=N6dwl5marHo&feature=youtu.be 

•  US  Military  Academy 
http://vimeo.com/27850108 
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Appendix  D:  RT-19  Student  and  Faculty  Posters 


Electronic  copies  of  student  and  faculty  posters  are  posted  on  the  Stevens  Sakai  project  site  at  the 
following  URL:  gateway.stevens.edu/home.html:  Proposal  &  Reports  -  RT19  2010-2011/RT19  Final 
Report  files 
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Appendix  E:  Comparison  of  Program  Components 

The  following  table  illustrates  the  various  elements  and  promising  practices  present  in  each  of  the  SE  Capstone  courses 
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